C example doesn't work outside nix-shell on macOS

[YorikSar]

Describe the bug
Example in examples/toolchains/cc fails if I run it with bazel run --config=nix :hello:

 % bazel run --config=nix :hello
INFO: Invocation ID: e8a0d447-f8a8-42cd-874a-30563a79614b
INFO: Analyzed target //:hello (21 packages loaded, 77 targets configured).
INFO: Found 1 target...
ERROR: /Users/tweag/workspaces/bazel_on_macos/rules_nixpkgs/examples/toolchains/cc/BUILD:3:10: Compiling hello.cc failed: (Exit 1): cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics ... (remaining 24 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
gcc is not installed
Target //:hello failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.783s, Critical Path: 0.06s
INFO: 4 processes: 4 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

To Reproduce

cd examples/toolchains/cc
bazel run --config=nix :hello

Expected behavior
We shouldn't expect user to always run examples in nix-shell, but rather they should work with system Bazel and ensure reproducibility with Nix via rules_nixpkgs, so this should work the same as in nix-shell environment.

Environment

• OS name + version: macOS 12.2.1
• Version of the code: master at aeeabee

Additional context
It seems examples don't actually use nixpkgs_config_cc provided by rules_nixpkgs, they use local_config_cc instead. It looks like this is because of missing flag --incompatible_enable_cc_toolchain_resolution in config (see bazelbuild/bazel#7260). We should probably turn it on in examples and test our code outside nix-shell as well.

[YorikSar]

Note that currently we test in CI:

• --config=nix in nix-shell with only Bazel declared in shell.nix files (5.x for Java example, 4.x for all others), but it also pulls in other nix-shell stuff like compiler (that makes my error go away)
• without --config=nix outside nix-shell which uses Bazel from environment (5.x for both Ubuntu and macOS in GitHub Actions images) that uses compiler from environment and doesn't break with gcc is not installed for some reason.

It seems like by these tests we imply that:

• rules_nixpkgs version can only be used in nix-shell environment (without --pure), which is not true, it works fine outside (if configured properly)
• if we don't use --config=nix, we can run "without Nix", which is not true, because rules_nixpkgs in WORKSPACE will run nix-build anyway

It looks like GitHub Actions provide Bazelisk, so we might want to switch to pinning Bazel version using Bazelisk and letting rules_nixpkgs fetch everything that's needed if --config=nix is specified.

@aherrmann @layus ping

[aherrmann]

It looks like GitHub Actions provide Bazelisk, so we might want to switch to pinning Bazel version using Bazelisk and letting rules_nixpkgs fetch everything that's needed if --config=nix is specified.

I think we should focus our efforts on two use-cases:

1. Nix provides Bazel and perhaps other development tools AND rules_nixpkgs imports system dependencies from Nix into Bazel.
   The Bazel derivation in nixpkgs includes patches for improved hermeticity in the context of Nix and compatibility with Nix and nixpkgs, so it is preferable in the rules_nixpkgs use-case to a vanilla Bazel distribution. Users who already want to use Nix to provide system dependencies into Nix are very likely to also want, or benefit from, Nix providing developer tools like Bazel as well.
   An open source example of this use case can be found in the daml repository. Whether the project uses a Nix-shell or some other means to provision developer tools from Nix (lorry, direnv, dev-env, etc.) is secondary.
2. Vanilla Bazel AND "bindist" or Bazel built system dependencies. I.e. no Nix involved at all.
   This use case is only relevant to rules_nixpkgs in so far as it needs to stay out of the way and not prevent this kind of use-case. The primary motivation are cross-platfrom projects that build on Unix (where Nix is available) and Windows (where Nix is practically not available), e.g. daml again. The secondary motivation are rule-sets (rules_haskell in particular) that need to function in both a Nix and a non-Nix setting and therefore need to be tested in both configurations.
   Whether vanilla Bazel is provided by bazelisk, download, or system package manager is secondary.

These combinations should be well tested in CI, if they are not in some cases, then this is a bug and needs to be fixed.
And, yes, this also includes covering more Bazel versions, something that we don't currently do, but should do.

Other combinations:

• Nix provided Bazel without rules_nixpkgs - this is irrelevant to rules_nixpkgs since it is not involved.
• Vanilla Bazel with rules_nixpkgs - it is unclear to me why this would be preferable over Nix provided Bazel with rules_nixpkgs.
  As such, I feel there are other areas where our effort would be better invested.

---

As for rules_nixpkgs's behavior in unsupported configurations. It's not so easy to determine automatically which configuration Bazel is run in. There are tricks like checking whether nix-build is in PATH or whether IN_NIX_SHELL (exact spelling might differ) is set, but, inevitably these are not complete. Therefore, rules_nixpkgs relies on use configuration. In particular, on users setting a Bazel platform that includes the @rules_nixpkgs_core//constraints:support_nix constraint. Note, --config=nix or similar is something that is specific to each individual project's .bazelrc configuration. The configuration in .bazelrc is not inherited through rule-set dependencies.

[evertedsphere]

@YorikSar can you try this now after the changes of #286? It may be that the .bazelrc change is sufficient to fix the issue, but I don't have a macOS device at hand to try.

[benradf]

@YorikSar are you able to check whether this issue still occurs after @evertedsphere's changes?