Protocol Buffer definition

[AntoinePrv]

The mof.json data format is a great initiative, but is currently only available to Julia users.

I believe it could be a good option to create a language agnostic Protocol Buffers declaration.
With it,

• it be possible to provide a parser in different languages (C++, Python...),
• it would be possible to read and write optimization problems as mof.json, but also in a compressed binary formats.

This is only a hot take, not something I have completely thought through, but I'm curious of what you think.

[mlubin]

I understand protocol buffers pretty well and would be happy to debate their use, but first I'm unclear about why JSON isn't sufficient. MathOptFormat has a JSON schema (see https://jump.dev/MathOptFormat/) and is intended to be language agnostic. We do want people to use it in other programming languages. A JSON parser in any language should work. For binary formats, we've used gzipped mof.json and the file sizes are slightly larger than gzipped MPS (the format is much more verbose). Could you explain more about the issues with JSON?

[AntoinePrv]

Yes, we can parse json in any language, but if I want to create/edit them in other languages, don't I have to write some boiler plate to validate the schema (like all the tokens, all the fields...), or make risk making silent typos in the token names that would end up skipping some data?
On the other hand, if I serialize into a C++ object, I get much more type safety.

[odow]

There are JSON-schema parsers in multiple languages http://json-schema.org. We took care to ensure that MOF is not Julia-specific.

Here was the previous time this was discussed: jump-dev/MathOptFormat#3.

In particular, https://developers.google.com/protocol-buffers/docs/techniques#large-data

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.

[mlubin]

There are a number of validators you can use in python or C++: http://json-schema.org/implementations.html#validators. There are some libraries for code generation from the schema: http://json-schema.org/implementations.html#generators-from-schemas. These are a bit lacking compared with code generation from protos.

The size limits of protocol buffers is indeed a fatal limitation. They can't support problems with more than 2^31 variables or constraints. Protocol buffer serialization typically breaks for files larger than 2GB.

If you really want the code generation, one option I could see is writing a proto3 schema for MathOptFormat but continuing to use JSON for serialization. You can read and write proto3 structs using JSON: https://codeburst.io/protocol-buffers-part-3-json-format-e1ca0af27774.

[AntoinePrv]

Thanks @odow @mlubin . I didn't know json schema could be validated automatically, I thought json schemas were more of a documentation kind of thing.
It seems that there also exists tools to convert json shemas to native classes, so that addresses all my concerns.