Fails to compile protobuf-src = "1.1.0+21.5"

[markmandel]

Not 100% sure if this is an issue with rust-linux-darwin-builder, but I also figure now that https://github.com/tokio-rs/prost 0.11 requires a local protoc, this is likely going to come up in the near future for a few people.

So we're attempting to use https://crates.io/crates/protobuf-src to vendor protoc locally as a build-dependency in our Cargo.toml, which generally works fine, except when we're trying to cross compile to a mac build, the following happens when it tries to compile:

➜  build git:(pr/update-deps) make build-macos-binary
docker run --rm -v /home/mark/workspace/quilkin:/workspace -w /workspace \
        -v ~/.cargo/registry:/root/.cargo/registry \
        -e "CARGO_TARGET_DIR=/workspace/target/build-image" \
        -e "CC=o64-clang" -e "CXX=o64-clang++" \
        us-docker.pkg.dev/quilkin/ci/rust-linux-darwin-builder:"1.65.0" \
                sh -c "rustup target add x86_64-apple-darwin && cargo build --release --target x86_64-apple-darwin"
info: syncing channel updates for '1.65.0-x86_64-unknown-linux-gnu'
info: latest update on 2022-11-03, rust version 1.65.0 (897e37553 2022-11-02)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
info: installing component 'rustfmt'
info: downloading component 'rust-std' for 'x86_64-apple-darwin'
info: installing component 'rust-std' for 'x86_64-apple-darwin'
   Compiling libgit2-sys v0.12.26+1.3.0
   Compiling libz-sys v1.1.8
   Compiling protobuf-src v1.1.0+21.5
   Compiling rustix v0.36.4
   Compiling axum-core v0.3.0
   Compiling addr2line v0.17.0
   Compiling serde_json v1.0.89
   Compiling miniz_oxide v0.5.4
   Compiling eyre v0.6.8
   Compiling kube-client v0.76.0
   Compiling backtrace v0.3.66
   Compiling regex v1.7.0
   Compiling stacker v0.1.15
   Compiling axum v0.6.1
   Compiling proc-macro-error v1.0.4
   Compiling psm v0.1.21
   Compiling darling v0.14.1
   Compiling darling v0.13.4
   Compiling crossbeam-utils v0.8.11
   Compiling ahash v0.8.0
   Compiling ahash v0.7.6
   Compiling tonic-build v0.8.4
   Compiling backoff v0.4.0
   Compiling regex-automata v0.1.10
   Compiling prost v0.11.3
   Compiling async-stream-impl v0.3.3
   Compiling derivative v2.2.0
   Compiling object v0.29.0
   Compiling snap v1.0.5
   Compiling os_str_bytes v6.3.0
   Compiling prometheus v0.13.3
   Compiling cached_proc_macro_types v0.1.0
   Compiling matchit v0.7.0
   Compiling same-file v1.0.6
   Compiling sync_wrapper v0.1.1
error: failed to run custom build command for `protobuf-src v1.1.0+21.5`

Caused by:
  process didn't exit successfully: `/workspace/target/build-image/release/build/protobuf-src-d5aea2da6cc4b31d/build-script-build` (exit status: 101)
  --- stdout
  running: "sh" "-c" "exec \"$0\" \"$@\"" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/protobuf-src-1.1.0+21.5/protobuf/configure" "--prefix=/workspace/target/build-image/release/build/protobuf-src-a2a5be1d214dd3cd/out/install" "--disable-shared" "--enable-static" "--disable-maintainer-mode"
  checking whether to enable maintainer-specific portions of Makefiles... no
  checking build system type... x86_64-pc-linux-gnu
  checking host system type... x86_64-pc-linux-gnu
  checking target system type... x86_64-pc-linux-gnu
  checking for a BSD-compatible install... /usr/bin/install -c
  checking whether build environment is sane... yes
  checking for a race-free mkdir -p... /bin/mkdir -p
  checking for gawk... no
  checking for mawk... mawk
  checking whether make sets $(MAKE)... yes
  checking whether make supports nested variables... yes
  checking whether UID '0' is supported by ustar format... yes
  checking whether GID '0' is supported by ustar format... yes
  checking how to create a ustar tar archive... gnutar
  checking whether make supports nested variables... (cached) yes
  checking for gcc... o64-clang
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether the compiler supports GNU C... yes
  checking whether o64-clang accepts -g... yes
  checking for o64-clang option to enable C11 features... none needed
  checking whether o64-clang understands -c and -o together... yes
  checking whether make supports the include directive... yes (GNU style)
  checking dependency style of o64-clang... gcc3
  checking whether the compiler supports GNU C++... yes
  checking whether o64-clang++ accepts -g... yes
  checking for o64-clang++ option to enable C++11 features... none needed
  checking dependency style of o64-clang++... gcc3
  checking how to run the C preprocessor... o64-clang -E
  checking for gcc... gcc
  checking whether the compiler supports GNU C... (cached) yes
  checking whether gcc accepts -g... yes
  checking for gcc option to enable C11 features... (cached) none needed
  checking whether gcc understands -c and -o together... (cached) yes
  checking dependency style of gcc... (cached) gcc3
  checking how to run the C preprocessor... gcc -E
  checking how to run the C++ preprocessor... o64-clang++ -E
  checking for g++... g++
  checking whether the compiler supports GNU C++... (cached) yes
  checking whether g++ accepts -g... yes
  checking for g++ option to enable C++11 features... (cached) none needed
  checking dependency style of g++... (cached) gcc3
  checking how to run the C++ preprocessor... g++ -E
  checking for stdio.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for strings.h... yes
  checking for sys/stat.h... yes
  checking for sys/types.h... yes
  checking for unistd.h... yes
  checking for wchar.h... yes
  checking for minix/config.h... no
  checking whether it is safe to define __EXTENSIONS__... yes
  checking whether _XOPEN_SOURCE should be defined... no
  checking for ar... ar
  checking the archiver (ar) interface... ar
  checking C++ compiler flags...... use user-supplied: -O0 -ffunction-sections -fdata-sections -fPIC --target=x86_64-unknown-linux-gnu
  checking for o64-clang++ options needed to detect all undeclared functions... cannot detect

  --- stderr
  configure: error: in `/workspace/target/build-image/release/build/protobuf-src-a2a5be1d214dd3cd/out/install/build':
  configure: error: cannot make o64-clang++ report undeclared builtins
  See `config.log' for more details
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 1

  build script failed, must exit now', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/autotools-0.2.5/src/lib.rs:715:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:150: build-macos-binary] Error 101

I'm sorta at a loss, so figured I'd ask here, since I'm figuring other people might have also run into the same thing? Or I can also take this over to the protobuf github? not sure. Help? 😞

[markmandel]

Created a gist with the config.log that was output.
https://gist.github.com/markmandel/0f5c270c80f138d2ca7be63d18f0f201

I am wondering if this is the issue:

configure:5664: $? = 0
configure:5708: result: yes
configure:5728: checking for o64-clang++ option to enable C++11 features
configure:5743: o64-clang++  -c -O0 -ffunction-sections -fdata-sections -fPIC --target=x86_64-unknown-linux-gnu  conftest.cpp >&5
In file included from conftest.cpp:20:
In file included from /usr/local/osxcross/target/bin/../SDK/MacOSX12.3.sdk/usr/include/c++/v1/exception:80:
/usr/local/osxcross/target/bin/../SDK/MacOSX12.3.sdk/usr/include/c++/v1/cstddef:44:15: fatal error: 'stddef.h' file not found
#include_next <stddef.h>

Why would stddef.h not be found? 🤔

[markmandel]

Might also be related to tpoechtrager/osxcross#211 🤔

[markmandel]

Okay! So I worked out what was going on, and made a workaround.

So.... what the prost needs inside rust-linux-darwin-builder image is a linux version of protoc to be able to generate the protobuf output.

But the compiler that is set in rust-linux-darwin-builder is the osxcross one... which fails to compile protoc for a variety of reasons -- and even if it did work, it wouldn't be possible to run it anyway, since it's for the wrong platform.

So my workaround is to have this setup.sh script run before I ask cargo to do compile my app:

set -eo pipefail

mkdir -p /opt/protoc
pushd /opt/protoc
apt-get update
apt-get -y install unzip
curl "https://github.com/protocolbuffers/protobuf/releases/download/v21.5/protoc-21.5-linux-$(uname -m).zip" -L -o protoc.zip
unzip protoc.zip
popd
/opt/protoc/bin/protoc --version
rustup target add x86_64-apple-darwin

...and have a feature that is enabled by default to vendor (which I turn off), so that my system will use this local protoc, rather than the vendored library when cross compiling to a mac.

I'm not 100% sure if it makes sense to have protoc built into rust-linux-darwin-builder as a handy toolchain to have, since other people might run into similar things?

I'll leave this here and see if other people run into this fun issue.

[joseluisq]

Sorry for the late response. Great that you sorted out the protobuf-src issue.

Since protobuf is a popular tool. I'm thinking that it can be useful for others to have it incorporated (image) in a similar way to zlib, libpq, etc rather than have to build it manually.

I mean, open to receiving a feature PR introducing protoc as a dependency if you think so.
We only have to make sure that builds for linux/macos (amd64/arm64). For example, a prost build test could be useful. 😅

[markmandel]

I mean, open to receiving a feature PR introducing protoc as a dependency if you think so.

Yeah, I'm not sure. Let's see who else has the issue? That was what I figured.

We only have to make sure that builds for linux/macos (amd64/arm64). For example, a prost build test could be useful.

The good thing is, protoc just generates rust code, so there isn't any compilation to worry about 😄