Making docs optional again

[MJDSys]

See as docs are required again due to #903, can we track what's necessary to make them optional again here? As one of my laptops doesn't have space for the docs (16G disk :( ), I'd be interested in helping fix the relevant issues. The two mentioned issues I've seen are:

• #901, which would require a useful message/action when docs are requested and not present
• #891, which would install the docs by default to help new people
• #1006 Add support for default installed optional features to manifest

Are there any others? And if you prefer to track this information elsewhere, please close this bug and point me in the right direction!

[vks]

Another problem are network file systems. For me it takes minutes to install the docs, the rest finishes almost instantly.

[pravic]

I'd prefer to leave docs as optional since I use the online version only and don't want to waste disk space or increase rustup update time.

[brson]

The major missing feature is that the manifest format needs to be extended to understand optional features: #1006.

[Boscop]

Yes, please make docs optional again! We work a lot with Raspberry Pis where space is very limited!

[wagenet]

This would certainly be nice for CI.

[Boscop]

And when updating a nightly or when different projects use different nightlies (with a rust-toolchain file) it always takes a long time to install/update them, it would be much faster without the docs.

[pravic]

The major missing feature is that the manifest format needs to be extended to understand optional features

Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).

The problem is (among others) that $toolchain/share/doc contains nearly 15000 files which is a big pressure on disk during toolchain updates - on Windows it takes a really significant amount of time to install rust-docs.

From #903:

It seems that most people are comforted knowing docs are available by default, so I want to turn them back on, then add some more options to remove them.

I am fine with default (though I really doubt that those who really need docs can't install them as a component), but let's make them as opt-out, so rustup would remember during the next update that rust-docs doesn't need to be downloaded.

[Boscop]

I can confirm that it takes a long time to install the docs on windows, and I'm dealing with multiple nightlies for different projects (with rust-toolchain files)..

But I'd prefer if the docs were opt-in, because most people only use the online docs. An internet connection is required to be able to use Rust anyway..

[Arnavion]

The major missing feature is that the manifest format needs to be extended to understand optional features

Is it still? Because now we have plenty of optional components in rustup (rust-std-*, rust-src, clippy-preview, etc).

@pravic The issue is that the manifest does not have a way express opt-out components, only opt-in.

[shepmaster]

The playground docker images are also about 300MB larger than they need to be because of the docs (which I'm working on removing at the moment).

[Geobert]

Installation of docs makes rustup hang at the end of the install process: #1464

[pronebird]

It takes ages for us to install Rust on Travis/Windows. Any progress on sorting this out? Really delays builds by 10 minutes or more which is impossible to work in productive way without wasting 2 hours on CI each day.

[nrc]

I have a plan for profiles which will address this. I hope to implement in the next few weeks

[johnthagen]

@nrc I'm sure this is nothing new to you, but wanted to provide another data point. Building a "Hello World" executable on Travis CI takes 8x longer on Windows. The vast majority of the time is spent installing rust-docs.

https://travis-ci.org/johnthagen/min-sized-rust/builds/465623377

The profile work will have a great impact on CI. ❤️

[Kampfkarren]

@nrc What's the status on that? rust-docs takes minutes to install when I never use them.

[thedavidmeister]

@brson @nrc is there anything i can do to help progress this?

[brson]

Yeah @johnthagen it's all pretty lolworthy. Maybe there's no right answer. Installed by default but decompresed on first use might be a good compromise - presumably its not the download that hurts windows users but touching so many files. Plus options to not install docs at all perhaps.

Decompressing on first use does make the assumption that everybody access the toolchain docs via rustup (or maybe cargo?). I'm not sure that's true, whether its because somebody wants to use a different browser or because rustup doc can't launch a browser.

rustup could even consider rust-docs installed w/o even downloading the tarball - download and decompress on first use. I think that's possible. It would though make the network required for first view of the docs. Probably the size of rust-docs tarball is not so much we need to care.

Just a strawman:

• don't make rust-docs optional again, on the assumption the tarball is sufficiently small that the download time and space for storing it are reasonable for most CI purposes
• rustup stashes rust-docs tarball somewhere and marks it as installed but not decompressed
• rustup puts placeholder index.html's where the main index, book, reference, and std would be that indicates the docs need to be decompressed, here's how
• add rustup doc --decompress for good measure
• when the user runs rustup component add rust-docs but the docs are already installed but not decompressed, tell them so, and to run rustup doc --decompress

[pravic]

Just FYI, the docs tarballs are about 30 MB in tgz format. It's not nothing.

[pronebird]

I think it's way easier to not install rust-docs by default and tell folks to run rustup component add rust-docs if they want them. It must be a very small effort to opt-out the docs from being installed by default, not sure what causes this issue to linger here for years.

[lnicola]

I'd say don't install rust-docs by default, but prompt about it in rustup-init.

[johnthagen]

I'd say don't install rust-docs by default, but prompt about it in rustup-init.

With a short warning that on Windows installing rust-docs takes a long time, to help new Windows users.

[lzutao]

rust-doc should be installed by default and have an opt-in (command line argument) to not install it. Because when you need the offline doc (no internet), there's no way to get it.

[lnicola]

@lzutao

• how do you install Rust?
• how many toolchains do you usually have installed?

[lzutao]

@lnicola Installing rust-doc by default is the decision of Rust team.
All I want is to have an opt-in way to not install it, especially when used in CI.

[Nemo157]

alternative strawman design following what I believe the OP was based on:

• the rust-docs component is marked "optional but default" in the manifest (#1006)
• rustup install will by default install "optional but default" components
• --[no-]default-optional-features can be specified to rustup install to force enable/disable installing "optional but default" components for a specific installation
• rustup update will not install "optional but default" components that are not currently installed (maybe it should print a message about them?)
• rustup component remove will allow removing "optional but default" components from a toolchain
• default-optional-features = true/false can be added to ~/.rustup/settings.toml to change the default behaviour for "optional but default" components with rustup install
• rustup-init will prompt the user about setting default-optional-features when installing rustup

(EDIT: keeping the component compressed and decompressing on use seems like it could also be worth it, but personally I would first prefer to have the ability to opt-out completely for most of my toolchains, only keeping docs for nightly and not the half-dozen nightly-date installs I have, along with removing the overhead from CI).

[SimonSapin]

If I remember correctly the consensus lately is that we’d prefer the docs to be included by default because the first time someone might need a local copy is when they’re offline, and at that point it might be too late to download them. But we want the ability to disable them for scenarios like CI. Unfortunately the rustup manifest format does not support yet having a component that is optional but enabled by default, so this needs some implementation work.

Separately, the rustdoc team wants to find a way to ship a compressed archive that can be used in place, so that there isn’t a need to create so many files: https://paper.dropbox.com/doc/Rustdoc-2019-roadmap-meeting--AXcjEXi3H6gpiujVOgSaSdmEAg-of8FmD8ZbXNvxD4swa45s

[thedavidmeister]

all the options proposed so far are reasonable imo, they respect offline access (either always or after the first access) and achieve what is needed here

a 30MB dl is trivial on CI, where many GB typically gets pulled down in 10-20 seconds (e.g. restoring cache on travis), still should be fast enough to not appear broken on regular connections

what is possible to implement right now without major refactoring?

[SamWhited]

30MB … still should be fast enough to not appear broken on regular connections

Please try to remember that "regular connections" are not the same speed everywhere. 30MB is very large, and would take a significant amount of time or cost a significant amount of money in bandwidth in many places.

[thedavidmeister]

i live in australia, we have plenty of terrible internet connections here ;)

if downloading the docs is truly a dealbreaker in many places there are no options other than making it optional

we should not discuss anything else if we agree the bandwidth cost is unacceptable

that said, if 30mb is a significant cost i'm not sure how it would be possible to work with rust on a large project anyway given the size of all the crates that end up as dependencies...

[dwijnand]

I think @pravic's solution strikes the right balance:

I am fine with default (though I really doubt that those who really need docs can't install them as a component), but let's make them as opt-out, so rustup would remember during the next update that rust-docs doesn't need to be downloaded.

[thedavidmeister]

given the age of this issue, i am a little worried that we might "commit" to an ideal that requires a lot of yak shaving... then 12 months passes because nobody has the time/expertise to land what was "committed to", despite best intent

do we have a passable option that doesn't require heroic effort from anyone?

[dwijnand]

I was hoping to back able to ask @nrc tomorrow/Tuesday, at the Rustup WG meeting.

[thedavidmeister]

that sounds awesome

[faern]

Installing them by default, but being able to fully opt out of their download completely sounds like the only really clean solution to me. Everyone who don't really care will get them by default and have access to them when they are offline, just like now. But CI and those who care about their bandwidth and time it takes to install on Windows can just append something like --minimal and save a lot of time.

I think optional, opt-out, components make sense even outside the scope of docs. By default rustup is somewhat "batteries included", but there is still a way to only get the bare minimum to compile Rust programs. Really useful for build servers or more constrained environments.

[retep998]

There's basically two separate but related features here:

• Allowing components to be optional yet default, with an opt out to never download them in the first place.
• Allowing components to be downloaded without actually being installed. Switch the src and docs components to use this by default, since they take a long time to install on NTFS, but aren't necessary to get code building.

Either feature on their own is helpful, but having both of them will make everyone happy. People who don't want any bandwidth wasted at all can opt out, people who are suddenly without internet and realize they forgot to install the docs won't be screwed, and people installing Rust on Windows won't have to suffer through agonizing installation times.

[johnthagen]

Does anyone know, now that @nrc is leaving Mozilla, what will become of his implementation of this feature?

[dwijnand]

An initial, still WIP PR was sent last week intended to bring a solution to this issue: #1673.

[thedavidmeister]

maybe this got worse, or maybe travis got worse

now seeing intermittent build timeouts:

https://travis-ci.com/holochain/holochain-rust/jobs/184030443

maybe the docs have grown recently and we hit some threshold?

[SimonSapin]

You might be seeing rust-lang/rust#58849. Workaround at rust-lang/rust#58849 (comment).

[SimonSapin]

Ah sorry, the issue I just linked is about running the rustdoc tool but this thread and your comment is about installing the rust-docs component of a rustup toolchain. Never mind.

[andywswan]

it seems to me that no one but 0.1% of the community needs the docs by default, but I just can't think of what to do next? shouldn't we create a poll or something somewhere?

[kinnison]

Progress is being made, though slowly, on installation profiles to allow for this without the need for polling a subset of the community. All I can ask is that you remain patient.

[ashthespy]

@kinnison Any way the community could help? If there are smaller sub tasks, I don't mind trying my hand at it.

[kinnison]

There is a draft PR at #1673 which needs updating against the reorganisation of the codebase and such. I'm not sure how much more work will be needed after that, though @nrc will hopefully have something to say on what his plans are.