Skip to content

Commit

Permalink
Enable xz archives (#1284)
Browse files Browse the repository at this point in the history 
* Expand the universe of known shas to include the .xz archive shas.

I have explicitly added keys for the files with the extension explicitly
on the file key. The point of this is to (in a future change) allow
tar.xz archives to be used instead of tar.gz archives.

This change keeps compatiblity with the old keys that assume the tar.gz
archives. I don't know if this is something that we should drop in the future
or just not worry about.

* Enable xz archive support.

XZ archives are about half the size of the gz equivalents. This
improves the cold repo bootstrap case a fair amount.

Co-authored-by: Daniel Wagner-Hall <dwagnerhall@apple.com>
  • Loading branch information
@illicitonion
Wren Turkal and illicitonion committed Apr 26, 2022
1 parent f7cb22e commit 841fc6f
Show file tree
Hide file tree
Showing 3 changed files with 5,226 additions and 24 deletions.

4 comments on commit 841fc6f

@krasimirgg
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this broke the ability to use rules_rust with a nightly channel: in the following example, I have a local checkout of the rules_rust git repo over at /usr/local/google/home/krasimir/rust/krasimirgg-git/rules_rust synced to this commit and try to use it in a repository with a nightly version. The error says: Prefix "2022-04-01/rust-nightly-x86_64-unknown-linux-gnu/rustc" was given, but not found in the archive. Here are possible prefixes for this archive: "rust-nightly-x86_64-unknown-linux-gnu". This setup worked with the parent rules_rust commit.

% tail -n +1 WORKSPACE.bazel BUILD.bazel foo.rs
==> WORKSPACE.bazel <==
workspace(name = "examples")

local_repository(
    name = "rules_rust",
    path = "/usr/local/google/home/krasimir/rust/krasimirgg-git/rules_rust",
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_register_toolchains(
    edition = "2021",
    iso_date = "2022-04-01",
    version = "nightly",
)

==> BUILD.bazel <==
load("@rules_rust//rust:defs.bzl", "rust_library")

rust_library(
    name = "foo",
    srcs = ["foo.rs"],
)

==> foo.rs <==
pub fn foo() -> i32 { return 42 }
% bazel build :foo
INFO: Repository rust_linux_x86_64 instantiated at:
  /usr/local/google/home/krasimir/rust/whole_archive_example/yolo/WORKSPACE.bazel:12:25: in <toplevel>
  /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/repositories.bzl:122:14: in rust_register_toolchains
  /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
  /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/repositories.bzl:327:30: in rust_repository_set
Repository rule rust_toolchain_repository defined at:
  /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/repositories.bzl:194:44: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rust_linux_x86_64':
   Traceback (most recent call last):
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/repositories.bzl", line 159, column 43, in _rust_toolchain_repository_impl
                build_components = [load_rust_compiler(ctx)]
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/private/repository_utils.bzl", line 310, column 24, in load_rust_compiler
                load_arbitrary_tool(
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/private/repository_utils.bzl", line 570, column 20, in load_arbitrary_tool
                ctx.extract(
Error in extract: java.io.IOException: Error extracting /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64: Prefix "2022-04-01/rust-nightly-x86_64-unknown-linux-gnu/rustc" was given, but not found in the archive. Here are possible prefixes for this archive: "rust-nightly-x86_64-unknown-linux-gnu".
ERROR: /usr/local/google/home/krasimir/rust/whole_archive_example/yolo/WORKSPACE.bazel:12:25: fetching rust_toolchain_repository rule //external:rust_linux_x86_64: Traceback (most recent call last):
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/repositories.bzl", line 159, column 43, in _rust_toolchain_repository_impl
                build_components = [load_rust_compiler(ctx)]
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/private/repository_utils.bzl", line 310, column 24, in load_rust_compiler
                load_arbitrary_tool(
        File "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rules_rust/rust/private/repository_utils.bzl", line 570, column 20, in load_arbitrary_tool
                ctx.extract(
Error in extract: java.io.IOException: Error extracting /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64: Prefix "2022-04-01/rust-nightly-x86_64-unknown-linux-gnu/rustc" was given, but not found in the archive. Here are possible prefixes for this archive: "rust-nightly-x86_64-unknown-linux-gnu".
ERROR: /usr/local/google/home/krasimir/rust/whole_archive_example/yolo/BUILD.bazel:3:13: //:foo depends on @rust_linux_x86_64//:toolchain_for_x86_64-unknown-linux-gnu_impl in repository @rust_linux_x86_64 which failed to fetch. no such package '@rust_linux_x86_64//': java.io.IOException: Error extracting /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to /usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c1a89d3d11c8b61357f4336cda255fc8/external/rust_linux_x86_64: Prefix "2022-04-01/rust-nightly-x86_64-unknown-linux-gnu/rustc" was given, but not found in the archive. Here are possible prefixes for this archive: "rust-nightly-x86_64-unknown-linux-gnu".
ERROR: Analysis of target '//:foo' failed; build aborted: Analysis failed

@wt
Copy link
Contributor

@wt wt commented on 841fc6f Apr 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to look into this. Having said that, it may take some time. Can you please look into the generated known shas and see what is missing? Knowing that would help me. I suspect this may have to do with a difference between how nightly keys are generated vs the normal keys. The fix is probably in the shell script that generates the keys. It would be lovely if you could take a look and see if you could fix it.

Thanks,
wt

@wt
Copy link
Contributor

@wt wt commented on 841fc6f Apr 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could also be the part of the code that extracts the archive. Honestly, I am not sure why we don't use download_and_extract instead of manually downloading and separately extracting. I'd have to look into that.

@krasimirgg
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I've tracked it down to the extraction logic, sent #1302 to hopefully fix this.

Please sign in to comment.