Skip to content
This repository has been archived by the owner on Nov 18, 2021. It is now read-only.

cue: proto support #5

Closed
mpvl opened this issue Jan 30, 2019 · 4 comments
Closed

cue: proto support #5

mpvl opened this issue Jan 30, 2019 · 4 comments
Labels
FeatureRequest New feature or request good first issue Good for newcomers protobuf

Comments

@mpvl
Copy link
Contributor

mpvl commented Jan 30, 2019 •
edited

Tracking issue for adding protobuf support to CUE:

  • import command reads text and binary proto instance
  • import command reads proto definitions
  • export command supports the converse
  • core libs
@mpvl mpvl changed the title proto support cue: proto support Jan 30, 2019
@mpvl mpvl added the good first issue Good for newcomers label Jan 31, 2019
@mpvl mpvl added the FeatureRequest New feature or request label Apr 8, 2019
@mpvl mpvl modified the milestones: v0.0.1, v0.1.0 Apr 27, 2019
mpvl added a commit that referenced this issue Jul 1, 2019
@mpvl
encoding/protobuf: add Builder API
93e9597 
A Builder parses a collection of .proto file and
organizes them according to a CUE package
layout. The exising Parse function is expressed
in this new API.

Issue #5

Change-Id: I980e7654d2b666dd1c637ad6d80f513096907a0b
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/2364
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
mpvl added a commit that referenced this issue Jul 1, 2019
@mpvl
encoding/protobuf: don't make oneOf required by default
ffe4ced 
This often makes sense to do, but it shouldn't be
the default.

Issue #5

Change-Id: I9f05ac9ee8a770489e7885da9c82790f091aee8b
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/2372
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
mpvl added a commit that referenced this issue Jul 1, 2019
@mpvl
encoding/protobuf: don't mark a single enum value as default
eae20e0 
Issue #5

Change-Id: Icb6e8b5c6c490ea7123e554a9fb86a2b8d8ba624
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/2373
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
mpvl added a commit that referenced this issue Jul 1, 2019
@mpvl
encoding/protobuf: don't map google.protobuf types
f9bd63e 
(at least not yet)

Issue #5

Change-Id: If8430cf8ec9bb376bcf0c69ec385f1f0d066c4e6
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/2374
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
@JohnAZoidberg
Copy link

What's the status of encoding Cue values as Protobuf?
I'd like to describe my data in cue and dump is as binary using Protobuf.
Should I go through JSON and convert that to Proto3?

@mpvl
Copy link
Contributor Author

mpvl commented Aug 14, 2020

It is not implemented so far. Should be straightforward, but stabilizing the new evaluator has priority.

But open to contributions.

@mpvl mpvl removed this from the v0.3.0-evaluator-rewrite milestone Aug 14, 2020
@mpvl mpvl added the protobuf label Feb 18, 2021
@mvdan
Copy link
Contributor

mvdan commented Mar 30, 2021

In case you're not aware, the new protobuf module has low-level primitive packages for encoding and decoding, for example for arbitrary data in protowire: https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/encoding/protowire

https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/encoding/prototext is message-based instead of token-based, though, so it's unclear if you could use it at all without a schema.

cueckoo pushed a commit that referenced this issue Apr 2, 2021
@mpvl
cmd/cue/cmd: simplify parseArgs in preparation of proto support
13abf25 
The parseArgs code had some pretty complicated logic.
This now separates out the cases for normal instances
and user-selected files to avoid at least one level of
comingling.

The goal of this refactoring is to ultimately allow schema
and value files to be separated early on, which, in turn,
will allow schema values to be computed early.
Such schema will then allow value formats to be interpreted
by the type of which they will ultimately land in CUE.

This is necessary to parse protobuf, which cannot be parsed
without a schema.

Issue #5

Change-Id: I2046948a8cf9bc5425ed437247dc89416d0fff76
cueckoo pushed a commit that referenced this issue Apr 2, 2021
@mpvl
cmd/cue/cmd: parseArgs: split values and schemas early
0dc7f24 
This will allow schema information to be used for the
parsing of values.

Issue #5

Change-Id: I7ab2bfb56666ba392b19274bae59da0152e16d9a
cueckoo pushed a commit that referenced this issue Apr 6, 2021
@mpvl
internal/encoding: pass schema to config
99023e9 
The idea here is as follows:
When unifying a bag of CUE and non-CUE, CUE can
function as a template for non-CUE. For instance, an
`int` CUE field could allow JSON string to be interpreted
as a number (as is the case for Protobuf).

Note that for Protobuf this is imperative, as a proto
buffers cannot be parsed without a schema.

Issue #5

Change-Id: I9a70c2e8489e6f7f3c26ee90c2fdb1d65ab6e473
cueckoo pushed a commit that referenced this issue Apr 6, 2021
@mpvl
cmd/cue/cmd: compute schema before values
478ebe3 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
cueckoo pushed a commit that referenced this issue Apr 8, 2021
@mpvl
cmd/cue/cmd: simplify parseArgs in preparation of proto support
893f45d 
The parseArgs code had some pretty complicated logic.
This now separates out the cases for normal instances
and user-selected files to avoid at least one level of
comingling.

The goal of this refactoring is to ultimately allow schema
and value files to be separated early on, which, in turn,
will allow schema values to be computed early.
Such schema will then allow value formats to be interpreted
by the type of which they will ultimately land in CUE.

This is necessary to parse protobuf, which cannot be parsed
without a schema.

Issue #5

Change-Id: I2046948a8cf9bc5425ed437247dc89416d0fff76
cueckoo pushed a commit that referenced this issue Apr 8, 2021
@mpvl
cmd/cue/cmd: parseArgs: split values and schemas early
079b92f 
This will allow schema information to be used for the
parsing of values.

Issue #5

Change-Id: I7ab2bfb56666ba392b19274bae59da0152e16d9a
cueckoo pushed a commit that referenced this issue Apr 8, 2021
@mpvl
internal/encoding: pass schema to config
b511e20 
The idea here is as follows:
When unifying a bag of CUE and non-CUE, CUE can
function as a template for non-CUE. For instance, an
`int` CUE field could allow JSON string to be interpreted
as a number (as is the case for Protobuf).

Note that for Protobuf this is imperative, as a proto
buffers cannot be parsed without a schema.

Issue #5

Change-Id: I9a70c2e8489e6f7f3c26ee90c2fdb1d65ab6e473
cueckoo pushed a commit that referenced this issue Apr 8, 2021
@mpvl
cmd/cue/cmd: compute schema before values
978c5b0 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
cueckoo pushed a commit that referenced this issue Apr 8, 2021
@mpvl
cmd/cue/cmd: preparse command line path expressions
c80bf0b 
This simplifies the transition to extracting schema
for dynamic context.

Issue #5

Change-Id: I3848a7f18f13aacaf989c889f22ea8cfde66ee5f
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
cmd/cue/cmd: simplify parseArgs in preparation of proto support
add266a 
The parseArgs code had some pretty complicated logic.
This now separates out the cases for normal instances
and user-selected files to avoid at least one level of
comingling.

The goal of this refactoring is to ultimately allow schema
and value files to be separated early on, which, in turn,
will allow schema values to be computed early.
Such schema will then allow value formats to be interpreted
by the type of which they will ultimately land in CUE.

This is necessary to parse protobuf, which cannot be parsed
without a schema.

Issue #5

Change-Id: I2046948a8cf9bc5425ed437247dc89416d0fff76
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
cmd/cue/cmd: parseArgs: split values and schemas early
7c00b64 
This will allow schema information to be used for the
parsing of values.

Issue #5

Change-Id: I7ab2bfb56666ba392b19274bae59da0152e16d9a
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
internal/encoding: pass schema to config
30890bd 
The idea here is as follows:
When unifying a bag of CUE and non-CUE, CUE can
function as a template for non-CUE. For instance, an
`int` CUE field could allow JSON string to be interpreted
as a number (as is the case for Protobuf).

Note that for Protobuf this is imperative, as a proto
buffers cannot be parsed without a schema.

Issue #5

Change-Id: I9a70c2e8489e6f7f3c26ee90c2fdb1d65ab6e473
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
cmd/cue/cmd: compute schema before values
31dd1d1 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
cmd/cue/cmd: preparse command line path expressions
fd77391 
This simplifies the transition to extracting schema
for dynamic context.

Issue #5

Change-Id: I3848a7f18f13aacaf989c889f22ea8cfde66ee5f
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
encoding/protobuf: always include type as second argument
ab73929 
This significanty simplifies finding the right mapping from
CUE to protobuf.

Also changed map type from map<A,B> to map[A]B. The former
caused issues as the comma caused it to be a separate
attribute argument (<> is not matched). Encosing it in
quotes was also annoying.

We considered using [A]:B, but using the map notation allows
for more extendibility and less ambiguity in case we must
support lists for some reason.

This is a backwards incompatible change.

Issue #5

Change-Id: Ief64d0c91d481fd7b03a4a8842faf377fd364a26
cueckoo pushed a commit that referenced this issue Apr 9, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
6d42f1a 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
cueckoo pushed a commit that referenced this issue Apr 11, 2021
@mpvl
cmd/cue/cmd: simplify parseArgs in preparation of proto support
75d0180 
The parseArgs code had some pretty complicated logic.
This now separates out the cases for normal instances
and user-selected files to avoid at least one level of
comingling.

The goal of this refactoring is to ultimately allow schema
and value files to be separated early on, which, in turn,
will allow schema values to be computed early.
Such schema will then allow value formats to be interpreted
by the type of which they will ultimately land in CUE.

This is necessary to parse protobuf, which cannot be parsed
without a schema.

Issue #5

Change-Id: I2046948a8cf9bc5425ed437247dc89416d0fff76
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9304
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
cueckoo pushed a commit that referenced this issue Apr 11, 2021
@mpvl
cmd/cue/cmd: parseArgs: split values and schemas early
8e5eeab 
This will allow schema information to be used for the
parsing of values.

Issue #5

Change-Id: I7ab2bfb56666ba392b19274bae59da0152e16d9a
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9305
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
cueckoo pushed a commit that referenced this issue Apr 11, 2021
@mpvl
internal/encoding: pass schema to config
8ba98ee 
The idea here is as follows:
When unifying a bag of CUE and non-CUE, CUE can
function as a template for non-CUE. For instance, an
`int` CUE field could allow JSON string to be interpreted
as a number (as is the case for Protobuf).

Note that for Protobuf this is imperative, as a proto
buffers cannot be parsed without a schema.

Issue #5

Change-Id: I9a70c2e8489e6f7f3c26ee90c2fdb1d65ab6e473
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9321
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
Reviewed-by: Paul Jolly <paul@myitcv.org.uk>
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
cmd/cue/cmd: compute schema before values
9d412ff 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
cmd/cue/cmd: preparse command line path expressions
3e4e439 
This simplifies the transition to extracting schema
for dynamic context.

Issue #5

Change-Id: I3848a7f18f13aacaf989c889f22ea8cfde66ee5f
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
encoding/protobuf: always include type as second argument
2f89a0b 
This significanty simplifies finding the right mapping from
CUE to protobuf.

Also changed map type from map<A,B> to map[A]B. The former
caused issues as the comma caused it to be a separate
attribute argument (<> is not matched). Encosing it in
quotes was also annoying.

We considered using [A]:B, but using the map notation allows
for more extendibility and less ambiguity in case we must
support lists for some reason.

This is a backwards incompatible change.

Issue #5

Change-Id: Ief64d0c91d481fd7b03a4a8842faf377fd364a26
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
0146467 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
encoding/protobuf/textproto: add encoder
9448fc3 
Doesn't really copy positioning and all comments, but
the textproto API is sort of hit or miss anyway whether
this information is supported at all.
So good enough for now.

Issue #5

Change-Id: Ia0d09a0c4b92756f68c2a09a114311b363fef33a
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
c5a5d47 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
cueckoo pushed a commit that referenced this issue Apr 12, 2021
@mpvl
encoding/protobuf/textproto: add encoder
962eb33 
Doesn't really copy positioning and all comments, but
the textproto API is sort of hit or miss anyway whether
this information is supported at all.
So good enough for now.

Issue #5

Change-Id: Ia0d09a0c4b92756f68c2a09a114311b363fef33a
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
cmd/cue/cmd: compute schema before values
5084de3 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
cmd/cue/cmd: preparse command line path expressions
a7f7adb 
This simplifies the transition to extracting schema
for dynamic context.

Issue #5

Change-Id: I3848a7f18f13aacaf989c889f22ea8cfde66ee5f
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
cmd/cue/cmd: compute schema before values
aaf6e84 
This allows the schema with which data files will
be merged before they are parsed. This is necessary
to support protobuf parsing, including Protobuf /
JSON mappings.

The usage of this property will be done in a followup CL.

Issue #5

Change-Id: I54b8826070217bd3d4e61d7d6f5b692f33553b12
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9322
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
cmd/cue/cmd: preparse command line path expressions
f228236 
This simplifies the transition to extracting schema
for dynamic context.

Issue #5

Change-Id: I3848a7f18f13aacaf989c889f22ea8cfde66ee5f
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9350
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
Reviewed-by: Paul Jolly <paul@myitcv.org.uk>
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
encoding/protobuf: always include type as second argument
9ff1e60 
This significanty simplifies finding the right mapping from
CUE to protobuf.

Also changed map type from map<A,B> to map[A]B. The former
caused issues as the comma caused it to be a separate
attribute argument (<> is not matched). Encosing it in
quotes was also annoying.

We considered using [A]:B, but using the map notation allows
for more extendibility and less ambiguity in case we must
support lists for some reason.

This is a backwards incompatible change.

Issue #5

Change-Id: Ief64d0c91d481fd7b03a4a8842faf377fd364a26
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
encoding/protobuf: always include type as second argument
0b74175 
This significanty simplifies finding the right mapping from
CUE to protobuf.

Also changed map type from map<A,B> to map[A]B. The former
caused issues as the comma caused it to be a separate
attribute argument (<> is not matched). Encosing it in
quotes was also annoying.

We considered using [A]:B, but using the map notation allows
for more extendibility and less ambiguity in case we must
support lists for some reason.

This is a backwards incompatible change.

Issue #5

Change-Id: Ief64d0c91d481fd7b03a4a8842faf377fd364a26
cueckoo pushed a commit that referenced this issue Apr 13, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
0e41c79 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
cueckoo pushed a commit that referenced this issue Apr 14, 2021
@mpvl
encoding/protobuf: always include type as second argument
3bdfa5d 
This significanty simplifies finding the right mapping from
CUE to protobuf.

Also changed map type from map<A,B> to map[A]B. The former
caused issues as the comma caused it to be a separate
attribute argument (<> is not matched). Encosing it in
quotes was also annoying.

We considered using [A]:B, but using the map notation allows
for more extendibility and less ambiguity in case we must
support lists for some reason.

This is a backwards incompatible change.

Issue #5

Change-Id: Ief64d0c91d481fd7b03a4a8842faf377fd364a26
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9368
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
Reviewed-by: Paul Jolly <paul@myitcv.org.uk>
cueckoo pushed a commit that referenced this issue Apr 15, 2021
@mpvl
encoding/protobuf/textproto: add encoder
dbca278 
Doesn't really copy positioning and all comments, but
the textproto API is sort of hit or miss anyway whether
this information is supported at all.
So good enough for now.

Issue #5

Change-Id: Ia0d09a0c4b92756f68c2a09a114311b363fef33a
cueckoo pushed a commit that referenced this issue Apr 15, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
c028026 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
cueckoo pushed a commit that referenced this issue Apr 15, 2021
@mpvl
encoding/protobuf/textproto: add encoder
841567d 
Doesn't really copy positioning and all comments, but
the textproto API is sort of hit or miss anyway whether
this information is supported at all.
So good enough for now.

Issue #5

Change-Id: Ia0d09a0c4b92756f68c2a09a114311b363fef33a
cueckoo pushed a commit that referenced this issue Apr 15, 2021
@mpvl
encoding/protobuf/textproto: add decoder implementation
4a288d5 
Note that this is incredibly buggy. Not much I can do,
as there just doesn't seem to be any good textproto parser
for Go, and this one is the recommended "gold standard".

Issue #5

Change-Id: Ieab0910dc4ea6072d9dc50e2947d8a7fb33ba7ef
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9369
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
cueckoo pushed a commit that referenced this issue Apr 15, 2021
@mpvl
encoding/protobuf/textproto: add encoder
a0035de 
Doesn't really copy positioning and all comments, but
the textproto API is sort of hit or miss anyway whether
this information is supported at all.
So good enough for now.

Issue #5

Change-Id: Ia0d09a0c4b92756f68c2a09a114311b363fef33a
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/9370
Reviewed-by: CUE cueckoo <cueckoo@gmail.com>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
@cueckoo cueckoo mentioned this issue Jul 3, 2021
Open
@cueckoo
Copy link

cueckoo commented Jul 3, 2021

This issue has been migrated to cue-lang/cue#5.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.

@cueckoo cueckoo closed this as completed Jul 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FeatureRequest New feature or request good first issue Good for newcomers protobuf
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mvdan @JohnAZoidberg @mpvl @cueckoo