New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Offline mode #5655

Open
matklad opened this Issue Jun 26, 2018 · 16 comments

Comments

Projects
None yet
6 participants
@matklad
Member

matklad commented Jun 26, 2018

Implementation PR: #4770
Original issue: #4686

Summary:

Running Cargo with -Z offline flag changes the resolution process such that it only considered crates, already downloaded locally.

Steps:

Stabilization TODO:

  • change -Z offline unstable cli option to a stable one, which works the same way as --frozen and --locked flags do.
  • #6014
@matklad

This comment has been minimized.

Member

matklad commented Jun 26, 2018

cc @Eh2406

@matklad matklad changed the title from offline mode to Offline mode Jun 26, 2018

@Eh2406

This comment has been minimized.

Contributor

Eh2406 commented Jun 26, 2018

Thanks for the write up. I think "no way to populate local crate storage" is overstated. There are pretty good, approximate, ways to do it like having used that dep in any previous project. Nead an ecosystem, build stdx, or equivalent and you have most of it. And complete, manually ways to do it, download all the crates and unpack them into carcos cache directory. It just takes a lot of disk space ~12 GB. Both of these the community can develop and improve once people are using offline. What is missing is a way to store a local compressed copy of crates and only uncompressed them as cargo needs them. That seems like it needs a lot of design work and could be an add on feature.

Given that, I think we should work to stabilize the existing offline, and come back to populateting local crate storage.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 27, 2018

One TODO item I might add as well is to audit the error messages and make sure they're understandable. For example if resolution fails due to the inability to update the registry then the error message should state that -Z offline was passed and removing it may allow the resolution to be fixed (if there's a missing version or something like that)

Otherwise I do personally still feel that population of the local storage is somewhat important. I see this as a sort of "what do I do before I get on a plane?" question. Or similarly what does Crater do before building all its crates? I think we'd want it along the lines of cargo cache stdx or something like that (aka just download a list of crates and all their transitive dependencies from crates.io) but I think we'd want to build that out in some command (the implementation shouldn't be too hard) before stabilizing.

@Eh2406

This comment has been minimized.

Contributor

Eh2406 commented Jun 28, 2018

An audit of the error messages is a good idea. I think the "-Z offline may be your problem" is in already. But the opposite would be good as well. "errore: no internet access; you can try -Z offline to try with what you have cashed"

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand. That would allow the community to experiment in the design space, before we have to commit to an implementation.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 28, 2018

But the opposite would be good as well

An excellent idea!

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand

I'd be totally on board with that

@luser

This comment has been minimized.

Contributor

luser commented Jul 12, 2018

I'm sure I mentioned this in a previous issue but I've used cargo-cacher for offline work before. It has a few options but I usually just run it with --all to fetch everything it can from crates.io. That always feels like overkill to me, though. Ideally I think I'd want some options like "fetch these specific crates and all their dependencies", "fetch the top N crates and all their dependencies", "fetch the newest version of every crate and all their dependencies".

@ehuss

This comment has been minimized.

Contributor

ehuss commented Jul 28, 2018

Is it intentional that offline mode will fail if an optional dependency is missing? It seems like it shouldn't because in online mode it doesn't fetch anything (AFAIK).

Example:

[package]
name = "off"
version = "0.1.0"

[dependencies]
# A very old version you are unlikely to have in your cache.
bitflags = {version="=0.1.0", optional=true}
> cargo +nightly build -Z offline
error: no matching version `= 0.1.0` found for package `bitflags`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 1.0.3, 1.0.2, 1.0.1, ...
required by package `off v0.1.0 (file:///Users/eric/Proj/rust/cargo/scratch/off)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising 
resolution failures, if this error is too confusing you may with to retry without the offline flag.
@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 28, 2018

@ehuss currently, yes, unfortunately. That's because lockfile generation requires the assumption that all possible features are enabled. Generating a lockfile without optional features is covered by #5133

@Jasper-Bekkers

This comment has been minimized.

Jasper-Bekkers commented Sep 5, 2018

A similar situation seems to happen with target specific dependencies. Something with this in the Cargo.toml will fail to build on windows with -Z offline:

[target.x86_64-unknown-linux-gnu.dependencies]
x11-dl = "~2.14"

Giving

error: no matching package named `x11-dl` found
@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Sep 5, 2018

@Jasper-Bekkers is that the full error message? Does Cargo print any other information?

@Jasper-Bekkers

This comment has been minimized.

Jasper-Bekkers commented Sep 8, 2018

It's pretty much the full error message.

>cargo build -Z offline
error: no matching package named `x11-dl` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `minifb v0.10.7`
    ... which is depended on by `voxelize-rs v0.1.0 (file:///...)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may with to retry without the offline flag.
@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Sep 9, 2018

@Jasper-Bekkers ah ok, did that not indicate enough that -Z offline was likely at fault?

@Jasper-Bekkers

This comment has been minimized.

Jasper-Bekkers commented Sep 9, 2018

-Z offline is indeed at fault; it's evaluating and matching Linux dependencies on a Windows machine while it shouldn't in similar vein to the broken optional dependencies. I think both should be made to work for this to be a useful workflow (eg. the airplane use case).

I figured I'd bump this but if a new ticket is needed then I can file a new one.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Sep 10, 2018

@Jasper-Bekkers oh that issue is different from this one and is covered by #5133

@Jasper-Bekkers

This comment has been minimized.

Jasper-Bekkers commented Sep 10, 2018

I don't think it does: #5133 (comment)

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Sep 10, 2018

That's an implementation bug, not a reflection of the feature-to-be-stabilized

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment