the maximum block size should be specified

[dvc94ch]

While currently a side effect of bitswap/libp2p, the main reason for introducing the concept of blocks is that they have a known finite size. An ipld library should be aware and check that the blocks are smaller than the maximum size. As a side note the cbor parser needs to check that the list size is within the block size before making allocations.

[mikeal]

While currently a side effect of bitswap/libp2p, the main reason for introducing the concept of blocks is that they have a known finite size. An ipld library should be aware and check that the blocks are smaller than the maximum size.

From IPLD’s point of view, there isn’t a max block size. When you use IPLD you may have block size limits imposed by the storage and transport, but those are opaque to us. Whatever these limits are, they should be checked and cause exceptions as soon as possible in the system imposing the limit.

As a side note the cbor parser needs to check that the list size is within the block size before making allocations.

It’s incredibly difficult to determine how large a dag-cbor block will be before you encode it, and doing so would be rather expensive (it’s a fully traversal and all of the logic you run during encode minus the encode allocations) compared to just throwing an exception after a block is encoded, assuming this only happens in 1 out of thousands of encodes.

For some block formats it’s much easier to predict the block size you’ll generate and the libraries for those formats should expose API’s to help with this (there’s one here in an experiment I’m doing) but there’s not much we can do generically or for dag-cbor, or dag-json for that matter.

[dvc94ch]

So the bitswap protocol should specify the max block size? It's an important thing that needs to be specified somewhere. Otherwise we may have implementations supporting different maximum block sizes unable to exchange blocks. I think the restriction is arbitrary, but there has to be one.

To have a single allocation encode, you could create a byte array with maximum block size. If the encoder tries to write out of bounds abort with exceeded block size. You can even put that array on the stack making it a zero allocation encode...

[rvagg]

It does seem like something that's in bitswap's area of responsibility. IPLD can produce arbitrarily large blocks and only depends on bitswap inasmuch as an application developer wants to couple the two things.

I know there's historical reasons that blocks end up with particular size ranges for IPFS, but I wouldn't mind seeing all of the issues around this laid out more clearly because I have trouble sometimes trying to work with the tradeoffs involved - larger blocks are more expensive to mutate and more expensive to decode if you only want partial data, but are cheaper to ship around large amounts of data and allow for more compact tree traversal when networks get involved. At a minimum it seems that IPLD could do with a discussion doc of some kind to talk about these tradeoffs (at the various layers, because they do differ) and the current state of play. I doubt we'll find reason to get to any firm restrictions though.

[mikeal]

So the bitswap protocol should specify the max block size? It's an important thing that needs to be specified somewhere.

I agree completely.

From most of the usage I’ve seen, once a block hits the storage layer, which is usually immediately after it’s encoded, it also gets handed to Bitswap and that’s when it should throw.

At a minimum it seems that IPLD could do with a discussion doc of some kind to talk about these tradeoffs (at the various layers, because they do differ) and the current state of play. I doubt we'll find reason to get to any firm restrictions though.

Agreed. We need to find better heuristics for this beyond just “block size” because it’s so hard to predict the block size. We have use cases all over where you want a map or list up to some size and at that point you want to use a multi-block data structure. But how do you write that?

All the times I’ve had to do this I’ve just set an arbitrary limit on the number of entires in a map or list before the code uses a multi-block structure instead, but that’s not a good predictor of the total encode size at all!

[dvc94ch]

I thought graphsync complements bitswap, but it looks more like a bitswap v3. Graphsync should also specify it's max block size...

[Stebalien]

Bitswap is easier to implement so most implementations should keep their bitswap implementation. IMO,

1. IPLD should specify a maximum recommended size.
2. Bitswap, Graphsync, etc. should link to this and say that they don't support blocks over the maximum recommended size.

[mikeal]

Whenever we’ve tried to come up with generic recommendations for block size we end up with a lot of “it depends 🤷‍♂️“ because we can all imagine situations where even relatively small block limits are a good idea and others where it would be perfectly acceptable to have incredibly large blocks.

I’ve been saying this for a while, but at some point we need to devote some time to writing up and proving something like a “CAP theorem for IPLD.” There’s a bunch of different vectors and, depending on your use case, you want to pull them in a particular direction and as you do you implicitly pull away from the other vectors. We all sort of know this exists in practice but we haven’t documented it sufficiently enough to make it useful to others.

When @rvagg started tackling the Collections docs we talked about this a lot. The settings you want to have for the shape of the tree you create isn’t just based on the size of the data and how you expect it to be read, the rate of mutation is also a factor because the tree shape changes the average block size and the number of blocks changed in every mutation — and that translates into both larger deltas during sync and more garbage to be collected in every node. It would be really nice if we had an elegant and well document model for how to talk about all of those tradeoffs.

[Stebalien]

Recommended maximum block size: 1MiB. This isn't a philosophical question, its a question of what's likely to work and what isn't. We have to pick a number and we should pick one that fits with the systems that currently use IPLD:

1. If all your blocks are under 1MiB, everything should "just work".
2. If your service/app can handle blocks up to 1MiB, it should "just work" with IPLD data.

Yes, there are complex design questions and trade-offs but this is just a recommendation. If you have a specific use-case and don't need to be completely portable, you can ignore it.

This is all users are asking for. A recommended size that will "just work" in most cases.

[mikeal]

I’m fine w/ 1m as the recommendation, but I don’t think we should enforce it in any of the codecs like we once did in ‘js-dag-cbor’.

…

-Mikeal

________________________________ From: Steven Allen <notifications@github.com> Sent: Monday, September 16, 2019 2:42 PM To: ipld/specs Cc: Mikeal Rogers; Comment Subject: Re: [ipld/specs] the maximum block size should be specified (#193) Recommended maximum block size: 1MiB. This isn't a philosophical question, its a question of what's likely to work and what isn't. We have to pick a number and we should pick one that fits with the systems that currently use IPLD: 1. If all your blocks are under 1MiB, everything should "just work". 2. If your service/app can handle blocks up to 1MiB, it should "just work" with IPLD data. Yes, there are complex design questions and trade-offs but this is just a recommendation. If you have a specific use-case and don't need to be completely portable, you can ignore it. This is all users are asking for. A recommended size that will "just work" in most cases. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#193?email_source=notifications&email_token=AAAAEQ27V4RD5EFQTVCF5UTQJ74VNA5CNFSM4IVYGSG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD62TOBA#issuecomment-531969796>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAAAEQYX5DM3ZI2RY4WUBSTQJ74VNANCNFSM4IVYGSGQ>.

[rvagg]

Seems reasonable to me for now. In my limited experience, as you edge toward 1M, everything but the "move this data around" operation tends to hurt more. Especially mutation but even decode/encode gets messy in different ways. There's got to be something about amortizing the mutation and encode/decode costs by interleaving with I/O costs that's involved in this CAP-like theorem.

Benchmarks, realistic test data, all of those things we really need and shouldn't put off too far into the future.