[RFC] Update .mrb file format

[masuidrive]

It's proposal of new .mrb file format.
Current .mrb file is hex format. It's too fat.

New .mrb file can have sections. now it have irep section only. I'll add debug section.
It have #880 in mind. The irep section has 'endianness' field and can choice big/little endian in the irep section.

I'm working on https://github.com/masuidrive/mruby/tree/binary

[matz]

I agreed with compact .mrb file.

I still think .mrb format should be endian neutral. There's no reason for "file format" to be endian sensitive, unless you REALLY want to read them through mmap. Only we need is in-memory correspondent of .mrb file, which should consume less memory.

FYI, I have a vague plan to remove irep array from mrb_state, instead, I'd add small irep arrays to each ireps.

[masuidrive]

.mrb loader support both endian.
So little endian CPU can load big endian mrb file.
'endianness' flag is for direct refer to ISEQ in .mrb file on ROM.

What do you think about ROM .mrb?
Should I need to create new section in .mrb file?

[mattn]

Why did you store compiler name/ver in irep section? I prefer to be upper-layer.

[mattn]

Also bytecode ver, endian things.

[matz]

In my opinion, ROM stored data format should be separated from .mrb file.

• mrb should be endian neutral, ROM is not
• mrb should be error tolerant (CRC e.g.), ROM need not to be.
• mrb should be read/write via I/O, ROM is not.

Design something that serves both purposes has no merit.

[skandhas]

I agree with what @mattn said. Compiler name/ver, bytecode ver and endianness are redundant in irep section. Those infomation can be stored in .mrb file header.

[miura1729]

I think that member whose size is 16 or 32bits should not assign on odd address. And in embedded system I think .mrb file needs reserved area for extending format. So I propose it need reserved member after endianess or top of IREP record 'B'.

[monaka]

I prefer @matz's opinion.
I believe there is the needs to ROMize.
But there are so many ROM-CPU connection type on real targets. So we mruby core team can't follow all of them.
In addition to @matz's exemplify: Alignments and endian is important for parallel ROMs. But they are not always important for serial ROMs like SPI connected.
(BTW, CRC check is required even if it is a programmable ROM, I think. ...back on topic.)

The solution is depends on the reason why binary format is required.
We can use compression if the reason is compact size.
(This may provide the another merit we can pack media contents with bytecodes.)

And I think we should provide a pluggable dump/load framework if the reason is ROM.

[monaka]

I agree with @mattn's saying basically.
It's useful to store {compiler name | version | bytecode ver. | endian} into the section header instead of the section binary. (And strictly I don't support adding endian to portable .mrb file format.)

[monaka]

How about to add file-format-type field in RITE file header?
If it exists, loader framework (this framework also "if it is exsits") can dispatch each reader subsystems.

Just a first plan:
file-format-type is uint32_t.
0 means traditional mrb format by @matz.
1 means new(?) mrb ascii hex format by @masuidrive and others.
2 means new(?) mrb binary format similar to 1.
3 to 255 is reserved for the future use.
256 - UINT32_MAX means free to use by application's self-responsibility.

[masuidrive]

@monaka
I think you don't need to file-format-type.
You can have application specific section in this file.

[monaka]

Ok. Go back to the root of this issue, then.

Could you tell us again why we need the new format?
For compaction? For archiving IREPs? For machine readability?

There have more than a merit. So it become easier to discuss if it was focused.

[masuidrive]

I moved compiler name/ver to file header.

But byte code ver still in irep section.
The file can contain some irep section what's different byte code ver.

I agreed remove endianness field.
I see sparately .mrb and ROM.

[masuidrive]

I think the new format is for extendable.
New format can contain data more than IREPs.

After that I'll work for containing debug information to the file.

[masuidrive]

Updated image https://creative.adobe.com/file/4b4b9a69-db93-4120-9cdf-38964b307042