New file format

[whitequark]

SolveSpace would benefit from a file format that suits the following required properties:

• compact, with a binary representation;
• has an alternate text representation, for tests;
• fast to read and write;
• easily extensible, while being backwards compatible;
• easily supports deeply nested structures, such as for hierarchical sketches.

In addition, an optional but desirable properties would be:

• easily readable and perhaps even writable from 3rd party software.

The current file format fulfills none of the above (not even the last requirement, since there is no explicit schema). I propose to replace it with a format based on Google Protocol Buffers, which has the following properties:

• the binary representation is naturally compact--it writes your data and generally does no special tricks during it, except that it defines a variable-length encoding for integers;
• the text representation is available alongside binary representation, and requires no manual work;
• has a highly optimized reader and writer implementation in C++;
• is literally designed because of a desire to handle protocol evolution well;
• is naturally nested;
• has a separate schema and very broad library support.

Such a format, with the initial version closely resembling the existing file format, seems like an ideal choice for us. The plan is to evolve it further by gradual extension, while remaining compatible with the old versions, as opposed to versioning; unknown entries are, generally, ignored, so with some care and testing, it is possible to retain an excellent degree of compatibility.

[whitequark]

@jwesthues Any objections? Disregard my earlier thoughts about using LLVM bitcode; after an in-depth discussion with LLVM folks that turned out to be a rather poor choice.

[whitequark]

The new file format should also use 64-bit handles (as would the in-memory representation). The rationale is that, once we make large assemblies fast, remapping of entities can rapidly exhaust the 65535-entity limit for one group, if one imports many instances of a large sketch. This migration is an ideal opportunity to add the 32→64 bit expansion code robustly.

[jwesthues]

That seems generally reasonable. It would be fashionable to use JSON or some other text representation today; but especially if you optimize assemblies to make better use of the GPU, a binary format's speedup seems material.

[whitequark]

Actually, the version 3 of Protocol Buffers has a canonical JSON serialization, so we could just use that. (Google generally recommends version 2, for reasons that I do not fully understand; but 3 is probably fine too.)

[whitequark]

It's also worth taking a look at Cap'n Proto, designed by the original author of Protocol Buffers while taking into account many of the downsides of the latter. It has some rather desirable properties, but a) completely lacks a human-readable format usable at runtime (in fact the author recommends the use of JSON in that case); b) lacks a bidirectional JSON parser and serializer; c) is not supported nearly as broadly. Overall I feel it's not the right choice, but it deserves a mention.

[Evil-Spirit]

The rationale is that, once we make large assemblies fast, remapping of entities can rapidly exhaust the 65535-entity limit

Actually when I implementing dxf import, I've easily found file which doesn't fit into 64k handles.

[Evil-Spirit]

The main problem is to build format which don't contains implementation details like remapping-lists. This format must be human-readable (in text form). I think it is good idea to have two sections:

1. The basic compact format representation - it is only reqests, groups and main parameters. This is fully-human readable and logically based. It must be sufficient to build whole sketch using only this part.
2. The precomputed part. This is for fast loading while building assemblies. By default this included into file, but can be avoided by user intention (for more compact files).

For the new-version migration we can just use 1) and don't bother about 2)

[jwesthues]

The main problem is to build format which don't contains implementation details like remapping-lists.

Without remapping, how do you propose to support nested assemblies?

[Evil-Spirit]

@jwesthues
Use some effective way to represent handle-chain path to entities. Like FullSubentPath used in AutoCAD.

[jwesthues]

You mean by making the entity's unique identifier variable-length?

[Evil-Spirit]

@jwesthues, yes. Since we want to have some nested sketches (assembly is one of the cases), we don't want to have one big heap of entities, so we can use variable-length identifiers.