Adding drake toolchain prototype

[adam-singer]

Description

Introduce drake toolchain prototype. This isn't intended for general use atm and should be considered prototype.

Fixes # (issue)

Type of change

Please delete options that aren't relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to
  not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please also list any relevant details for your test configuration

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[aaronmondal]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and 4 discussions need to be resolved

---

-- commits line 2 at r1:
nit: "Add drake toolchain configs" or "Add drake toolchain prototype"

---

tools/toolchain-drake/Dockerfile line 15 at r1 (raw file):

# limitations under the License.

FROM ubuntu:22.04 AS dependencies

This needs to be pinned so that we don't get a "Unpinned dependencies" score downgrade from the OpenSSF scorecard.

---

tools/toolchain-drake/Dockerfile line 23 at r1 (raw file):

# hadolint ignore=DL3003
RUN git -c http.sslVerify=false clone https://github.com/blakehatch/drake.git && \

Is there a specific reason that we need sslVerify=false? If we're missing certificates, would it make sense to add the cacerts package?

nit: Do we want to always use the latest commit, or should the checkout be pinned?
nit: We could use a shallow clone at a specific commit to speed up the clone. This would improve reproducibility and reduce clone time.

---

tools/toolchain-drake/toolchain-drake.sh line 56 at r1 (raw file):

# Get the sha256 value, this will fail due to empty string in the sha256 field.
set +o pipefail
SHA256_HASH=$(nix run .#nativelink-worker-toolchain-drake.copyTo docker://localhost:5001/nativelink-toolchain-drake:latest -- --dest-tls-verify=false 2>&1 | grep "got:" | grep -o 'sha256-[^[:space:]]*')

nit: Consider breaking this line.

[adam-singer]

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Vercel, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), pre-commit-checks, ubuntu-22.04, vale, and 2 discussions need to be resolved

---

-- commits line 2 at r1:

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

nit: "Add drake toolchain configs" or "Add drake toolchain prototype"

Done.

---

tools/toolchain-drake/Dockerfile line 15 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

This needs to be pinned so that we don't get a "Unpinned dependencies" score downgrade from the OpenSSF scorecard.

Done.

---

tools/toolchain-drake/Dockerfile line 23 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

Is there a specific reason that we need sslVerify=false? If we're missing certificates, would it make sense to add the cacerts package?

nit: Do we want to always use the latest commit, or should the checkout be pinned?
nit: We could use a shallow clone at a specific commit to speed up the clone. This would improve reproducibility and reduce clone time.

Done.

---

tools/toolchain-drake/toolchain-drake.sh line 56 at r1 (raw file):

Previously, aaronmondal (Aaron Siddhartha Mondal) wrote…

nit: Consider breaking this line.

Done.

[aaronmondal]

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: windows-2022 / stable

[adam-singer]

Reviewable status: 2 of 1 LGTMs obtained, and pending CI: windows-2022 / stable

[adam-singer]

Reviewed 1 of 1 files at r3, all commit messages.
Reviewable status: 2 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable

[adam-singer]

Reviewed 2 of 2 files at r2.
Reviewable status: 2 of 1 LGTMs obtained, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable

[zbirenbaum]

Reviewable status: 3 of 1 LGTMs obtained, and pending CI: Cargo Dev / macos-13, and 1 discussions need to be resolved

---

tools/toolchain-drake/toolchain-drake.sh line 67 at r3 (raw file):

# Patch flake.nix with sha256 value.
sed -i -E "s|sha256 = \"\"; # DO NOT COMMIT DRAKE SHA256 VALUE|sha256 = \"${SHA256_HASH}\"; # DO NOT COMMIT DRAKE SHA256 VALUE|" "${FLAKE_NIX_FILE}"

nit: Maybe it's just reviewable but it's really difficult to make out what is going on on these lines due to the comment. Could you clean this up a bit?

[adam-singer]

Reviewable status: 3 of 1 LGTMs obtained, and 1 discussions need to be resolved

---

tools/toolchain-drake/toolchain-drake.sh line 67 at r3 (raw file):

Previously, zbirenbaum (Zach Birenbaum) wrote…

nit: Maybe it's just reviewable but it's really difficult to make out what is going on on these lines due to the comment. Could you clean this up a bit?

This is a sed search and replace for a specific line. The problem we have is producing the toolchain is not within the reproducibility situation due to using the installer scripts provided by drake. As a compromise of not having to reverse eng their build dependencies at this time, we are generating the image, then applying a patch with the image hash to the flake.nix to generate the rest of the container layers. Also why this will be considered a prototype for now. The right solution would be to unroll explicitly all dependencies needed at their specific build/code points.