Add rust rules

[cpaelzer]

We do not yet have the rust toolchain in main, but AFAIU it is being worked on.
Since those rule extensions can lead to some discussions we want to start that early and that is what I'm trying to do here.
Please MIR-Team have a look, and once we are happy with it we will likely invite more people (e.g. foundations) to have a look if it matches their thoughts/plans.

[setharnold]

Thanks for starting this.

I've got a few concerns:

• it's possible that fixed dependencies may be new major versions and require significant porting work of the 'leaf' application in order to integrate them
• it's possible that this significant porting work may have been done in a way in the upstream projects that makes these changes "unwelcome" in an LTS or ESM release -- changes that some or many users would consider a regression, for example
• I believe the foundations team will periodically backport a newer rustc and cargo as they are required by Firefox, but this is a time-limited commitment on their part with the switch to snap-based Firefox that's supported by Mozilla rather than Canonical
• versions of projects that are adapted to newer dependencies may also require newer rustc or cargo and thus be impractical to support without backported toolchains.

The security team is not staffed (nor experienced) for the full scope of the problem in the full generality. If "upgrade the dependencies then rebuild the project" is as I expect, we may need significant effort from the owning teams to meet our collective goals of providing secure software without making too-drastic changes to our LTS and ESM releases. Is this process the right place to ask for shared maintenance assurances, similar to the new test plan assurances?

Thanks

[setharnold]

This isn't exactly the right place for this, but all the right people are here already; there's apparently some effort in the C++ community to try to make it more rust-like:

https://lists.debian.org/msgid-search/CAA8KeOLapdoe=SCyHsKnro6bWEk1s9JAJj703c55Xo-k_Z3Y=g@mail.gmail.com

[slyon]

Except for a small inline comment, this LGTM!

I agree that this puts lots of stress on the security team, keeping track of many non-packaged rust module sources and moving toolchains... How is this situation handled for golang packages that use static linking? It sounds very similar to me.

Foundations backporting new rustc/cargo toolchain periodically sounds like a very intrusive change. I guess we need to pull in some other Foundations folks who know better how the Rust toolchain is to be handled.

[cpaelzer]

Thanks for starting this.

I've got a few concerns:

All you bring up is valid @setharnold , but also all that you bring up already is exactly that way for golang packages that are in main with vendored sources nowadays.
So I'd assume we should use the chance to more clearly define expectations/commitment - but will not hold up rust as it isn't special/new in that regard.

I'd prefer to do an initial probe for opinions/preferences in the regular MIR-meeting and from there I can draft some additional rules to match what we came up.

[slyon]

I got a recommendation from within Foundations (cc @schopin-pro) that we might want to require Rust-built packages to export their lockfile in /usr/share/$pkg/, in order to keep track of the exact dependency versions. What do you think?

[cpaelzer]

I got a recommendation from within Foundations (cc @schopin-pro) that we might want to require Rust-built packages to export their lockfile in /usr/share/$pkg/, in order to keep track of the exact dependency versions. What do you think?

If that is easy and reliable enough to do that sounds like an easy and trivial rule, and it would be helpful like any other kind of manifest. @schopin-pro is there a common path to expect this file after build, a common command to get it?
Also I've read that if it is created might depend if it is an application or library, do we only want it in one of the cases - or do we need to force its creation for the others?

Furthermore for @setharnold I found that what he brought up was almost already covered. Just the explicit mention of "this kind of cases" as a rule and the explicit mention of "backports" also being expected from the owning Team. I have extended the rules accordingly - ready for a re-discussion later today.

[schopin-pro]

I got a recommendation from within Foundations (cc @schopin-pro) that we might want to require Rust-built packages to export their lockfile in /usr/share/$pkg/, in order to keep track of the exact dependency versions. What do you think?

If that is easy and reliable enough to do that sounds like an easy and trivial rule, and it would be helpful like any other kind of manifest. @schopin-pro is there a common path to expect this file after build, a common command to get it? Also I've read that if it is created might depend if it is an application or library, do we only want it in one of the cases - or do we need to force its creation for the others?

I'm going to clarify my proposal : any package that has code built from Rust sources (as opposed to the -dev packages which AFAICT ship the source tree) should ship the lockfile. This file is generated whenever the Rust code is built, no matter the type of crate, library or application. The difference lies on versioning: libraries are expected not to version their lockfile but applications are. For instance, I'm not expecting librust-lazy-static-dev to ship a lockfile, but ripgrep should.

You will usually find the Cargo.lock file next to the Cargo.toml file which is the Rust version of our debian/control file. In projects with multiple crates, the lockfile will be generated for the entire workspace and will be next to the toplevel Cargo.toml that defines that workspace.

The location should perhaps be /usr/share/doc/$pkg/ instead of /usr/share/$pkg ? A lockfile can easily get in the 10s of kB, and in extreme cases the MB realms, and removing /usr/share/doc is already supposed to be a semi-supported way to reduce the footprint of the OS (mentioned in the Debian policy).

[setharnold]

On Tue, Nov 09, 2021 at 03:38:21AM -0800, Lukas Märdian wrote: I got a recommendation from within Foundations (cc @schopin-pro) that we might want to require Rust-built packages to export their lockfile in /usr/share/$pkg/, in order to keep track of the exact dependency versions. What do you think?

This would be a vast improvement over the current use of /usr/.crates2.json that is currently preventing people from installing two or more rust packages at once :) ± apt-file search .crates2.json bat: /usr/.crates2.json cbindgen: /usr/.crates2.json difference: /usr/.crates2.json fd-find: /usr/lib/cargo/.crates2.json hexyl: /usr/.crates2.json lalrpop: /usr/.crates2.json lscolors: /usr/.crates2.json nitrocli: /usr/.crates2.json pulldown-cmark: /usr/.crates2.json ripgrep: /usr/.crates2.json rustdoc-stripper: /usr/.crates2.json rusty-tags: /usr/.crates2.json ucd-generate: /usr/.crates2.json xml-rs: /usr/.crates2.json https://bugs.launchpad.net/ubuntu/+source/rust-ripgrep/+bug/1868517

[cpaelzer]

Interim result from my check of Cargo.lock file
0. background info https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

1. our builds to not have the file at the moment
2. even if it is created dh-cargo would remove it https://salsa.debian.org/rust-team/dh-cargo/-/blob/master/cargo.pm#L145 currently
3. reason for that according to the changelog is ""Remove Cargo.lock since this prevents us from upgrading stuff". I'm not yet entirely sure what this is abour ... ?!
4. We might need to carry a delta to dh-cargo to not remove, and instead put it in the suggested dir (or re-run something like https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)

OTOH I just rebuilt a rust app with dh-cargo modified to not remove 'Cargo.lock' but still it isn't around after build, so there might be more needed to (always) create it.

But first I guess we need to inquire and understand what the Problem was that Debian had with this file. Maybe they are aware of an issue we are not seeing yet?

[slyon]

LGTM!

[didrocks]

I have a few requests/improvement needed (some semantic, some other maybe more oversights?)

On the .lock discussion, I’m not an expert on rust lock files, but they may be quite similar to the ones in the go worlds.

Basically, you have a go.mod file in go, which list all your direct dependencies, with the minimum required version of each.

On build, go builds a go.sum file. This list every dependencies (direct and transitive ones) with a checksum of downloaded version. This is what is used in go module "mode" (there is no vendor/ directory at this point).

That way, if someone else in your development team builds the same project, thanks to your go.mod file, the build tool redownload on a local private repository the same versions that you initally checked out. Thanks to MVS, the algorithm will ensure that all your direct and transitive dependencies are the same, and thus, lock them in some way. The .sum files contains the checksums of each project, so that you can be sure (this file is committed in the repo) that anyone redownloading and rebuilding the same project has non tempered dependencies (you then can’t retag an already released tag in one of your module, as the checksum would differ).

Then, when you ask to populate a vendor/ directory (mostly for distro, as the day to day go usage as a developer doesn’t need it), the modules are copied there (trimmed down only the packages that are used by your project). If there is any mismatch between go.mod and the vendor/ directory, then go build will heavily complain.

The right way to update something in the vendor directory is thus to update go.mod and regenerate the vendor/ directory with updated parts (or use a replace directive). However, this file should not be removed.

[schopin-pro]

Rust lockfiles are indeed very similar, with the nuance that there's no MVS algorithm, so the lockfile is the mechanism that ensure that everyone uses the same dependencies. When the lockfile is present, ©argo doesn't need to compute a new solution.

This is the reason why dh-cargo removes the lockfile: it forces the tooling to recompute a dependency tree, and thus picking up the Debian versions. Otherwise, your application will try to build using whatever was the version of the libraries when the upstream maintainer last ran cargo update. However, I still cannot figure out why the build doesn't generate a lockfile.

In any case, it should be possible to explicitly generate one using cargo generate-lockfile, which we could integrate to dh-cargo.

[setharnold]

Another feature of the Cargo.lock file is to allow building a project when some dependency crates have been yanked. The Rust ecosystem has a way for crate authors to 'unpublish' specific versions of crates so that that version is harder to use in a build. If you've got a Cargo.lock file, you can use it to retrieve specific versions of crates, even ones that have been yanked. If you only have a Cargo.toml file, it cannot be used to retrieve yanked versions of crates. This distinction may not make sense in the context of Ubuntu builders, when Internet access is not allowed in the first place. https://www.reddit.com/r/rust/comments/ctayew/rust_crypto_developers_please_stop_the_yanking/ is a very nice rant about this topic that might give some extra perspective about how it works. Having the exact versions used in the build will probably be very useful for "software bill of materials" requirements for selling to certain customers, beyond just being convenient for us to have this data exposed. Thanks

[cpaelzer]

...

This is the reason why dh-cargo removes the lockfile: it forces the tooling to recompute a dependency tree,

And now it makes sense to me why this is done in the configure stage of dh-cargo - Thanks!

In any case, it should be possible to explicitly generate one using cargo generate-lockfile, which we could integrate to dh-cargo.

cargo generate-lockfile was the right start, but it isn't that straight forward.
You need to also be aware of /usr/share/cargo/bin/cargo which is the wrapper that makes cargo behave the Debian-way.

I spare you the details of a long trip into dh-cargo, but the TL;DR is that it uses avoid-dev-deps which implies the non-creation of a Cargo.lock file.
This is done by the cargo wrapper here and we can not 100% rely on every package being able to do that (as extra downloads are prohibited).
Since - for now - we require all dependencies to be vendored anyway that isn't a problem for packages in main.

When calling generate-lockfile through the wrapper the argument avoid-dev-deps is not added (only for "build", "rustc", "doc", "test", "bench", "install"). Therefore calling generate-lockfile through the wrapper after build indeed gets us what we want.

Therefore I think we can require a Cargo.lock to be present and modify dh-cargo to create it.
If it is created ok then put it in the suggested path at /usr/share/doc/$pkg/.
If it fails, do not error out.

Until that is available (and we'd only do once we kind of agree on this) it can be done via:

#debian/install:
+Cargo.lock /usr/share/doc/mdevctl

#debian/rules:
 #!/usr/bin/make -f
+include /usr/share/dpkg/architecture.mk
+
 export DH_VERBOSE = 1
 
 export DEB_BUILD_MAINT_OPTIONS = hardening=+all
 
 %:
        dh $@ --buildsystem cargo
+
+override_dh_auto_build:
+       dh_auto_build
+       CARGO_HOME=./debian/cargo_home /usr/share/cargo/bin/cargo generate-lockfile

P.S. and putting it in doc and being text also makes it auto-gzipped.

[setharnold]

On Tue, Nov 16, 2021 at 05:33:27AM -0800, Christian Ehrhardt wrote: ``` #debian/install: +Cargo.lock /usr/share/doc/mdevctl #debian/rules: #!/usr/bin/make -f +include /usr/share/dpkg/architecture.mk + export DH_VERBOSE = 1 export DEB_BUILD_MAINT_OPTIONS = hardening=+all %: dh $@ --buildsystem cargo + +override_dh_auto_build: + dh_auto_build + CARGO_HOME=./debian/cargo_home /usr/share/cargo/bin/cargo generate-lockfile ```

Hmm, would this need to be done for every single Rust package in main? in Ubuntu? it'd be nice if we could modify our tooling to avoid these steps, so every rust package behaves the same. Otherwise it feels like a lot of deltas from Debian and a lot of inconsistency. Thanks

[cpaelzer]

> + CARGO_HOME=./debian/cargo_home /usr/share/cargo/bin/cargo generate-lockfile > ``` Hmm, would this need to be done for every single Rust package in main? in Ubuntu? it'd be nice if we could modify our tooling to avoid these steps, so every rust package behaves the same. Otherwise it feels like a lot of deltas from Debian and a lot of inconsistency.

Hi Seth, if you read all the above you'll see that the intention already was essentially "if we agree on this we will make our dh-cargo to do that automatically". The example just ensures that we are aware of the overall steps needed and allows anyone interested to experiment with it.

[setharnold]

On Tue, Nov 16, 2021 at 11:00:37PM -0800, Christian Ehrhardt wrote: if you read all the above you'll see that the intention already was essentially "if we agree on this we will make our dh-cargo to do that automatically". The example just ensures that we are aware of the overall steps needed and allows anyone interested to experiment with it.

Ah good, sorry I missed that bit. :( Thanks

[didrocks]

The changes LGTM, Let me approve this. Thanks for working on that!

[vorlonofportland]

Please require the toolchain to be fixed to use the Built-Using field in Ubuntu, instead of codifying this other field.

[cpaelzer]

The idea was to follow Debian which had a debate in Debian to separate classic Built-Using from X-Cargo-Built-Using/XB-X-Cargo-Built-Using, but if that is the preference that is ok.

I rechecked the current state and it seems we are even ok as-is right now.
It seems that nowadays we always get both values.
Currently dh-cargo generates the same content into both variables:
https://sources.debian.org/src/dh-cargo/28/dh-cargo-built-using/?hl=165#L165
And packages usually reference both in d/control:
https://sources.debian.org/src/rust-bindgen/0.59.1-2/debian/control/?hl=243#L242

That means that we should be able to just change the wording to the more common Cargo-Built-Using which also means that tools relying on it like the check for component mismatches would already work.

On the first few rust based MIRs we have to manually check that the assumption that all is properly reported via Built-Using is really true , I'll add a statement for that in the MIR Team check section. If we find a technical reason some are missed => bug, if we find common patterns that make them be missed => new rules.

Done:

• I updated the wording to use Cargo-Built-Using
• added a statement about the preliminary status of the rust rules
• I added a statement to double-check the correctness of rust Cargo-Built-Using

[schopin-pro]

Actually, the Built-Using field only contains the rustc version, while XB-Cargo-Built-Using contains all the crates used. There's a trailing _x in the code that's fairly easy to miss, I only went back because I looked at the ripgrep binary:

zstd -d < control.tar.zst
./0000755000000000000000000000000014113505654007715 5ustar  rootroot./control0000644000000000000000000000511514113505654011322 0ustar  rootrootPackage: ripgrep
Source: rust-ripgrep
Version: 13.0.0-2
Architecture: amd64
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net>
Installed-Size: 4155
Depends: libc6 (>= 2.34), libgcc-s1 (>= 4.2)
Built-Using: rustc (= 1.53.0+dfsg1+llvm-4ubuntu1)
Section: utils
Priority: optional
Multi-Arch: allowed
Homepage: https://github.com/BurntSushi/ripgrep
Description: Recursively searches directories for a regex pattern
 ripgrep is a line-oriented search tool that recursively searches your current
 directory for a regex pattern while respecting your gitignore rules and
 automatically skip hidden files/directories (smart filtering) and binary files.
 ripgrep is similar to other popular search tools like The Silver Searcher, ack
 and grep.
 .
 ripgrep is generally faster than both The Silver Searcher and GNU grep. It
 defaults to recursive directory search and won't search files ignored by your
 `.gitignore` files. Use ripgrep if you like speed, filtering by default, fewer
 bugs, and Unicode support.
 .
 On the other hand, if you like multiline search, then ripgrep may not quite
 meet your needs (yet), and it will never support fancy regex features such as
 backreferences or lookaround
X-Cargo-Built-Using: rust-aho-corasick (= 0.7.10-1), rust-atty (= 0.2.14-2), rust-base64 (= 0.13.0-1), rust-bitflags (= 1.2.1-1), rust-bstr (= 0.2.12-1), rust-bytecount (= 0.6.0-1), rust-byteorder (= 1.4.3-2), rust-cfg-if-0.1 (= 0.1.10-2), rust-cfg-if (= 1.0.0-1), rust-clap (= 2.33.3-1), rust-crossbeam-utils (= 0.8.5-1), rust-encoding-rs (= 0.8.22-1), rust-encoding-rs-io (= 0.1.6-2build1), rust-fnv (= 1.0.6-1), rust-globset (= 0.4.8-2), rust-grep-cli (= 0.1.6-2), rust-grep (= 0.2.8-2), rust-grep-matcher (= 0.1.5-2), rust-grep-pcre2 (= 0.1.5-1), rust-grep-printer (= 0.1.6-1), rust-grep-regex (= 0.1.9-2), rust-grep-searcher (= 0.1.8-2), rust-ignore (= 0.4.18-2), rust-itoa (= 0.4.3-1), rust-lazy-static (= 1.4.0-1), rust-libc (= 0.2.103-1), rust-log (= 0.4.11-2), rust-memchr (= 2.3.3-1), rust-memmap (= 0.7.0-1), rust-num-cpus (= 1.13.0-1), rust-once-cell (= 1.5.2-1), rust-pcre2 (= 0.2.3-1), rust-pcre2-sys (= 0.2.5-1), rust-regex-automata (= 0.1.8-2), rust-regex (= 1.3.9-1), rust-regex-syntax (= 0.6.25-1), rust-ryu (= 1.0.2-1), rust-same-file (= 1.0.6-1), rust-serde (= 1.0.130-1), rust-serde-json (= 1.0.41-1), rust-strsim (= 0.9.3-1), rust-termcolor (= 1.1.0-1), rust-textwrap (= 0.11.0-1build1), rust-thread-local (= 1.1.3-3), rust-unicode-width (= 0.1.8-1), rust-walkdir (= 2.3.1-1), rustc (= 1.53.0+dfsg1+llvm-4ubuntu1)
./md5sums0000644000000000000000000000060314113505654011234 0ustar  rootroot82cc2a1fdf7721bd88911cb638a71e22  usr/bin/rg
536668671f52e8e32f9f607aa4b70816  usr/share/doc/ripgrep/TODO.Debian
d613394d6c66ef2cd2ee0ed5fafcaa32  usr/share/doc/ripgrep/changelog.Debian.gz
f78c0f9ef0e769e9929d5e942a11ca02  usr/share/doc/ripgrep/copyright
ef4cf19ff3297de685a0ab8e6a12ecf1  usr/share/man/man1/rg.1.gz
7d274f4cd16b398d1b72173ebbf5c492  usr/share/zsh/vendor-completions/_rg

[cpaelzer]

Yeah @schopin-pro I remember seeing the same on my initial more thorough analysis - which along the discussion here got us to the rules we had, it even is in the resolved conversations of this PR :-). But this time missed that little _x - thanks!

I guess the question will have to be - do we:
a) change it back to X-Cargo-Built-Using?
b) do we file a bug against dh-cargo/rustc which needs to be resolved as part of the toolchain MIR to ensure this info lands in Built-using for Ubuntu *1.

@vorlonofportland - would you prefer and support such a post build mangling in the ubuntu builds to make normal Built-Using work?

*1 implementation probably needs to be in a different component since the packages refer to the field in their d/control it can't be fixed in dh-cargo but would need to be a post build mangling of the meta data. Probably as part of pkgbinarymangler or such.

[cpaelzer]

Oh BTW @schopin-pro that bug for Built-using (if we create it) would go along another one for the already discussed need to make our dh-cargo to allow for proper .lock files.

[schopin-pro]

I had a look at the Debian policy regarding Built-Using, and it is stated there that the purpose of the field is strictly for licensing issues. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921284 which is where the issue was raised for golang. There's been some inquiries from the Go side regarding migrating X-Cargo-Built-Using towards a more neutral field that could then be used by dh-golang as well, I'll keep an eye on the situation.

[cpaelzer]

...
I'll keep an eye on the situation.

Thank you!

[schopin-pro]

Based on feedback in the rustc MIR, I suggest including a requirement for the package to list all its vendored sources in a field XS-Vendored-Sources-$language, formatted as a comma delimited list of $package@$version items.
The reason for having this field language-specific is that each language has its own package namespace. For instance, you can find a "ipaddr" package in a lot of languages.

However, I'm yet to find a more satisfying mechanism than having the maintainer manually fill the fields themselves, alongside a sanity check in d/rules. Both the generation script and the check can of course live in dh-cargo for Rust.

Ideally we'd use debian/substvars for this, but apparently it's not taken into account by dpkg-source by default, which makes it even more awkward as the caller now has to figure out how to pass the correct options to whatever they're using for their package.

@setharnold I'd love your feedback on this?

[setharnold]

I'm not very familiar with the dh_ family of tools :(

I think anything that relies upon maintainer action / initiative is going to be doomed to be out of date. It really does need to be "free" for the packager; this seems like a problem the Debian golang and rust teams are interested in solving. I wonder if there's a good way for dh to provide most of the infrastructure for it, and try to simplify the language-specific portions?

This might be way off base though, as I said, I'm vastly unfamiliar with dh's internals.

[schopin-pro]

I think I need to expand a little bit on some details that I glossed over.

First, we're talking about tracking vendored dependencies, which is expected to be a requirement for Rust package in main, and is already the case for Go package AIUI. The Debian discussion is about other source packages that are statically linked into the binary, which is NOT the same thing.

Second, this is a source-level property of the package, and the generated data can be quite massive. This is why we cant it in the Sources index, which isn't downloaded by our users by default, hence the use of a XS field, which in turns led to my manual update predicament, as this means it cannot be part of the build process.

However, the whole point of having a sanity check in the debian/rules is to avoid the data going stale. Thinking about it some more, the symbols files work the same way, with the packager having to do a manual update but is guided by the tooling. The idea is indeed to leverage the packaging tools (i.e. dh-cargo for Rust) to automate this. Of course, the first implementation of the field being rustc, which has custom packaging, I'll do it explicitly in the debian/rules there. I still need to start working on vendoring support altogether in dh-cargo anyway ;-).

[jawn-smith]

I'd love to also have vendoring support in dh-golang. The Debian tooling for golang predates golang's modules design and therefore predates a formal notion of vendoring. Due to this we end up packaging the source code for tons of 'vendored' requirements in separate packages. As an example we have golang-github-shirou-gopsutil. It is a package that ships no binaries, only source code. This is used to build another package, syncthing, that builds actual binaries. Changing this over to a dh-golang supported vendoring design would be great.

[schopin-pro]

To be fair, regardless of golang formal notion of vendoring, the Debian tooling would have sought to split vendored dependencies in separate packages, as that's deemed the most maintainable way from a security PoV (patch once, rebuild everything). However, we revert this for at least some of our packages, e.g. runc which has a ton of golang deps in Debian but instead vendors them in Ubuntu.

What I'm proposing here is just formalizing how we track those vendored dependencies so that we know what needs patching, and once there's agreement on the format adding support for it in the languages tooling.