Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vendor protoc #62

Open
dunmatt opened this issue Aug 18, 2022 · 10 comments
Open

Vendor protoc #62

dunmatt opened this issue Aug 18, 2022 · 10 comments
Labels
good first issue Good for newcomers help wanted Extra attention is needed

Comments

@dunmatt
Copy link

dunmatt commented Aug 18, 2022

prost-build has the "vendored" feature so that I need not have protoc installed. How hard would it be to add a similar feature to pbjson-build?

@tustvold
Copy link
Contributor

tustvold commented Aug 18, 2022

I think you should be able to just enable the vendored feature flag for prost-build and things will work? Ultimately pbjson-build just requires a descriptor file, it is agnostic to how it is produced and doesn't directly interact with protoc at all.

@dunmatt
Copy link
Author

dunmatt commented Aug 18, 2022

Hmmm, my first attempt to do it that way didn't work, I'll do some more digging, it may well be user error.

@dunmatt
Copy link
Author

dunmatt commented Aug 18, 2022

The problem seems to be in pbjson-types:

error: failed to run custom build command for `pbjson-types v0.4.0`

Caused by:
  process didn't exit successfully: `/home/matt/code/comb3/seen_schema/seen-rs/target/debug/build/pbjson-types-53d2f1897182c455/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=/home/matt/.cargo/registry/src/github.com-1ecc6299db9ec823/pbjson-types-0.4.0/protos/google/protobuf/types.proto

  --- stderr
  thread 'main' panicked at '
  Could not find `protoc` installation and this build crate cannot proceed without
  this knowledge. If `protoc` is installed and this crate had trouble finding
  it, you can set the `PROTOC` environment variable with the specific path to your
  installed `protoc` binary.

  For more information: https://docs.rs/prost-build/#sourcing-protoc
  ', /home/matt/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.11.1/src/lib.rs:1227:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

@aierui
Copy link

aierui commented Aug 19, 2022

I also met

    Checking mockito v0.31.0
   Compiling pbjson-types v0.4.0
error: failed to run custom build command for `pbjson-types v0.4.0`

Caused by:
  process didn't exit successfully: `/Users/develop/aierui/influxdb_iox/target/debug/build/pbjson-types-ee8401d0b4f56f87/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-changed=/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos/google/protobuf/types.proto
  Running: "/Users/yixiao/.rpc-tools/protoc" "--include_imports" "--include_source_info" "-o" "/Users/develop/aierui/influxdb_iox/target/debug/build/pbjson-types-7a84daf192de4c6f/out/proto_descriptor.bin" "-I" "/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos" "/Users/yixiao/.cargo/registry/src/rsproxy.cn-8f6827c7555bfaf8/pbjson-types-0.4.0/protos/google/protobuf/types.proto"

  --- stderr
  Error: Custom { kind: Other, error: "protoc failed: google/protobuf/any.proto: File not found.\ngoogle/protobuf/api.proto: File not found.\ngoogle/protobuf/descriptor.proto: File not found.\ngoogle/protobuf/duration.proto: File not found.\ngoogle/protobuf/empty.proto: File not found.\ngoogle/protobuf/field_mask.proto: File not found.\ngoogle/protobuf/source_context.proto: File not found.\ngoogle/protobuf/struct.proto: File not found.\ngoogle/protobuf/timestamp.proto: File not found.\ngoogle/protobuf/type.proto: File not found.\ngoogle/protobuf/wrappers.proto: File not found.\ngoogle/protobuf/types.proto:5:1: Import \"google/protobuf/any.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:6:1: Import \"google/protobuf/api.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:7:1: Import \"google/protobuf/descriptor.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:8:1: Import \"google/protobuf/duration.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:9:1: Import \"google/protobuf/empty.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:10:1: Import \"google/protobuf/field_mask.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:11:1: Import \"google/protobuf/source_context.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:12:1: Import \"google/protobuf/struct.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:13:1: Import \"google/protobuf/timestamp.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:14:1: Import \"google/protobuf/type.proto\" was not found or had errors.\ngoogle/protobuf/types.proto:15:1: Import \"google/protobuf/wrappers.proto\" was not found or had errors.\n" }
warning: build failed, waiting for other jobs to finish...

➜ influxdb_iox git:(main) protoc --version
libprotoc 3.13.0

@tustvold
Copy link
Contributor

tustvold commented Aug 20, 2022

Aah, yeah I can see that would be a problem. I would be very happy to review a PR adding this functionality, and get it released, otherwise I can probably find time in the latter part of next week to look into this. I'm afraid I am a little bit underwater at the moment 😅

@tustvold tustvold added help wanted Extra attention is needed good first issue Good for newcomers labels Aug 20, 2022
@aierui
Copy link

aierui commented Aug 21, 2022

For my problem, can set the following env to solve it.

export PROTOC=/usr/local/bin/protoc
export PROTOC_INCLUDE=/usr/local/include/google/protobuf

@rekby
Copy link

rekby commented Sep 11, 2022

I have similar problem - after update prost-build and pbjson - I can't compile crate without protoc.

It is not problem for me personal, but I want transparent compile my crate on end customer computer without installed protoc.

env vars can't solve the case

@rekby
Copy link

rekby commented Sep 11, 2022

I can research about solve the protoc (in background, not fast). But I have trouble with build the crates on my laptop:
https://gist.github.com/rekby/34f5e8b072f6eddc5fc18dfd0f124821

Is any prepare need to start work with the repo?

@tustvold
Copy link
Contributor

tustvold commented Sep 12, 2022

Unfortunately this is really a prost issue, not a pbjson issue. I recommend checking out tokio-rs/prost#657 which has some background on the various changes that led to this predicament

As far as solutions go, if you wish for users to not need protoc, you will need to not use a build.rs and instead vend the generated code. This can easily be done by making build.rs into a binary, and then running this. Alternatively you could check out https://github.com/neoeinstein/protoc-gen-prost

Is any prepare need to start work with the repo?

This does not appear to be related to pbjson or even prost, it looks like something is messed up with your build toolchain. Perhaps try re-installing Rust, or perhaps raise an issue on anyhow, as that appears to be where the failure is "error: could not compile anyhow due to 56 previous errors".

@notmgsk
Copy link

notmgsk commented Oct 14, 2022

Sort of related: would you be open to not requiring protoc in the pbjson_types crate? Seems like it would be simple enough just to version the generated code in the crate rather than have to build it?

rnarubin added a commit to standard-ai/pbjson that referenced this issue Oct 14, 2022
This change moves pbjson-type's code generation to a manual executable
instead of build.rs. Now the generated files are checked into the repo,
and users do not need `protoc` to build the pbjson-types crate.

This has a slight drawback in that pbjson developers will need to invoke
the generator manually after any relevant code-gen changes. This should
generally be a worthwhile tradeoff, as several users have encountered
trouble with requiring protoc (see influxdata#62).

That drawback could potentially be averted with some CI steps to run the
generator and check-in the code automatically; this change does not
include such automation however.
rnarubin added a commit to standard-ai/pbjson that referenced this issue Oct 14, 2022
This change moves pbjson-type's code generation to a manual executable
instead of build.rs. Now the generated files are checked into the repo,
and users do not need `protoc` to build the pbjson-types crate.

This has a slight drawback in that pbjson developers will need to invoke
the generator manually after any relevant code-gen changes. This should
generally be a worthwhile tradeoff, as several users have encountered
trouble with requiring protoc (see influxdata#62).

That drawback could potentially be averted with some CI steps to run the
generator and check-in the code automatically; this change does not
include such automation however.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants