Size alignment

[illiliti]

Is there a builder API to append zero padding to buffer, so it becomes n-byte aligned?

Such API would be nice to have in order to stack up flatbuffers in buffer and then advance between them without needing to do memmove or copying.

Currently I'm appending padding myself, but I'd really need to know if builder can do this alone
What I do right now(all checks are omitted because it is example):

unsigned char tmp_buf[1024] = {0};
size_t padding = (-len) & 0x7; // 8 byte alignment
memcpy(tmp_buf, msg, len);
unsigned char zero[8] = {0};
memcpy(tmp_buf + len, zero, padding);
len = len + padding;
msg = tmp_buf;

But it is inefficient and rather hacky since it requires copy. I'd like to reuse builder buffer

[mikkelfj]

Well, it would require a bit of digging, but I belive there almost is, though your approach works just as well, if you polish a bit: When you get the final buffer you can either get an aligned allocated buffer, or you can ask for the size, and then provide a buffer to get the internal buffer state copied into. Since you have the size, you can just align upwards before copying. Generally, you would also want to know the alignment of the start of buffer, and there is a call for that too. You should find all of the in the flatcc_builder.h file with comments. If you need help I could try to dig further.

See

void *flatcc_builder_copy_buffer(flatcc_builder_t *B, void *buffer, size_t size);
size = flatcc_builder_get_buffer_size(B);
uint16_t flatcc_builder_get_buffer_alignment(flatcc_builder_t *B);

I think, there is probably also a call to set the final buffer size before the above happens.

In the 0.6.2 changelog entry:

Pad top level buffer end to largest object in buffer

there is on fix where the final buffer is zero padded to the alignment size of the buffer start, which was not originally the case because it was easy to retro fit when needed, but Googles flatc tool did pad to a multiple so I added that. The first formal release to support that is only a few days old in the; 0.6.2 and 0.6.3 releases, but the change itself was added somewhat before that release. It is probably worth tracking down that commit to see exactly what was done since you can probably do the same.

With this change the buffer alignment should automatically pad to the alignment size. I think you can manually control the minimum aliignment with:

flatcc_builder_ref_t flatcc_builder_create_buffer(flatcc_builder_t *B,
        const char identifier[FLATBUFFERS_IDENTIFIER_SIZE],
        uint16_t block_align,
        flatcc_builder_ref_t ref, uint16_t align, flatcc_builder_buffer_flags_t flags);
 int flatcc_builder_start_buffer(flatcc_builder_t *B,
        const char identifier[FLATBUFFERS_IDENTIFIER_SIZE],
        uint16_t block_align, flatcc_builder_buffer_flags_t flags);

but the end paddihg only takes effect in the latest release as per the above change log entry.

In principle you can probably find a low level technique to set the alignment later, e.g. by directly writing to the builder state struct before finalizing, but that would be a hack. There are complications with nested buffers to be aware of.

[mikkelfj]

The padding commit is here:

5362443

The critical change is in: align_buffer_end

which is called by flatcc_builder_create_buffer directly, or flatcc_builder_end_buffer indirectly as it internally calls flatcc_builder_create_buffer.

Thus, setting the block_align parameter during buffer_start/create should have the desired effect.

[illiliti]

Nope, the above code does not always work. Neither start/create_buffer works. I still get unaligned size sometimes...

[illiliti]

#391 this works better for me

[mikkelfj]

Direct buffer only works for small buffers and only with default emitter.

[illiliti]

My buffer is only 500~ bytes, i guess it is small enough

[mikkelfj]

Maybe, but a fix for flatcc needs to work universally.

[mikkelfj]

It is possible block align only affects regular content. Flatcc by default places vtables and might not be affected. There are numerous header comments, and builder doc, which I have not read recently. I suggest digging there first. Also, see monster_test.c.

Are you suggesting that start buffer does not work since you had to export the set block align function?

I don’t think it can be public because it must appear from the beginning. Setting a large block align could potentially waste a lot of space and might be the wrong tool, not sure.

As for bug, maybe it goes away if you call finalize aligned?

I don’t recall all details anymore.

I don’t think as_root is the right place, it should be in buffer extraction, and, by default the buffer already should be padded to internal alignment even if it might require a fix.

I suggest you dig more if interested, and I’ll help with guidance and PR if necessary.

[illiliti]

Are you suggesting that start buffer does not work since you had to export the set block align function?

No no, it works. Its api is just more verbose than set block align function

As for bug, maybe it goes away if you call finalize aligned?

Nope, incorrect size is still present even with finalize aligned.

I don’t think as_root is the right place, it should be in buffer extraction, and, by default the buffer already should be padded to internal alignment even if it might require a fix.

Agree.

I suggest you dig more if interested, and I’ll help with guidance and PR if necessary.

Thanks a lot. I'll certainly do, as I'm going to depend on this lib, just need to finish my project first