rustdoc for liblibc is 2.3GB large

[alexcrichton]

The libc crate generates documentation on a continuous basis and right now the gh-pages branch weighs in around 2.3 GB (!!!).

This apparently is so large that GitHub doesn't render all the documentation. This was a recent-ish regression I believe, although I don't quite know what.

cc @nrc do you know of any architectural changes here?

[bluss]

The same files are present many times over:

find . -name "W_OK*"
./i686-unknown-linux-gnu/libc/W_OK.v.html
./i686-unknown-linux-gnu/libc/notbsd/W_OK.v.html
./i686-unknown-linux-gnu/libc/unix/W_OK.v.html
./i686-unknown-linux-gnu/libc/unix/notbsd/W_OK.v.html
./x86_64-unknown-freebsd/libc/freebsdlike/W_OK.v.html
./x86_64-unknown-freebsd/libc/W_OK.v.html
./x86_64-unknown-freebsd/libc/bsd/freebsdlike/W_OK.v.html
./x86_64-unknown-freebsd/libc/bsd/W_OK.v.html
./x86_64-unknown-freebsd/libc/unix/freebsdlike/W_OK.v.html
./x86_64-unknown-freebsd/libc/unix/W_OK.v.html
./x86_64-unknown-freebsd/libc/unix/bsd/freebsdlike/W_OK.v.html
./x86_64-unknown-freebsd/libc/unix/bsd/W_OK.v.html
./x86_64-unknown-netbsd/libc/netbsdlike/W_OK.v.html
./x86_64-unknown-netbsd/libc/W_OK.v.html
./x86_64-unknown-netbsd/libc/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-netbsd/libc/bsd/W_OK.v.html
./x86_64-unknown-netbsd/libc/unix/netbsdlike/W_OK.v.html
./x86_64-unknown-netbsd/libc/unix/W_OK.v.html
./x86_64-unknown-netbsd/libc/unix/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-netbsd/libc/unix/bsd/W_OK.v.html
./x86_64-unknown-openbsd/libc/netbsdlike/W_OK.v.html
./x86_64-unknown-openbsd/libc/W_OK.v.html
./x86_64-unknown-openbsd/libc/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-openbsd/libc/bsd/W_OK.v.html
./x86_64-unknown-openbsd/libc/unix/netbsdlike/W_OK.v.html
./x86_64-unknown-openbsd/libc/unix/W_OK.v.html
./x86_64-unknown-openbsd/libc/unix/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-openbsd/libc/unix/bsd/W_OK.v.html
./arm-linux-androideabi/libc/W_OK.v.html
./arm-linux-androideabi/libc/notbsd/W_OK.v.html
./arm-linux-androideabi/libc/unix/W_OK.v.html
./arm-linux-androideabi/libc/unix/notbsd/W_OK.v.html
./arm-unknown-linux-gnueabihf/libc/W_OK.v.html
./arm-unknown-linux-gnueabihf/libc/notbsd/W_OK.v.html
./arm-unknown-linux-gnueabihf/libc/unix/W_OK.v.html
./arm-unknown-linux-gnueabihf/libc/unix/notbsd/W_OK.v.html
./x86_64-unknown-dragonfly/libc/freebsdlike/W_OK.v.html
./x86_64-unknown-dragonfly/libc/W_OK.v.html
./x86_64-unknown-dragonfly/libc/bsd/freebsdlike/W_OK.v.html
./x86_64-unknown-dragonfly/libc/bsd/W_OK.v.html
./x86_64-unknown-dragonfly/libc/unix/freebsdlike/W_OK.v.html
./x86_64-unknown-dragonfly/libc/unix/W_OK.v.html
./x86_64-unknown-dragonfly/libc/unix/bsd/freebsdlike/W_OK.v.html
./x86_64-unknown-dragonfly/libc/unix/bsd/W_OK.v.html
./x86_64-unknown-linux-musl/libc/W_OK.v.html
./x86_64-unknown-linux-musl/libc/notbsd/W_OK.v.html
./x86_64-unknown-linux-musl/libc/unix/W_OK.v.html
./x86_64-unknown-linux-musl/libc/unix/notbsd/W_OK.v.html
./aarch64-unknown-linux-gnu/libc/W_OK.v.html
./aarch64-unknown-linux-gnu/libc/notbsd/W_OK.v.html
./aarch64-unknown-linux-gnu/libc/unix/W_OK.v.html
./aarch64-unknown-linux-gnu/libc/unix/notbsd/W_OK.v.html
./mips-unknown-linux-gnu/libc/W_OK.v.html
./mips-unknown-linux-gnu/libc/notbsd/W_OK.v.html
./mips-unknown-linux-gnu/libc/unix/W_OK.v.html
./mips-unknown-linux-gnu/libc/unix/notbsd/W_OK.v.html
./x86_64-unknown-linux-gnu/libc/W_OK.v.html
./x86_64-unknown-linux-gnu/libc/notbsd/W_OK.v.html
./x86_64-unknown-linux-gnu/libc/unix/W_OK.v.html
./x86_64-unknown-linux-gnu/libc/unix/notbsd/W_OK.v.html
./i686-apple-darwin/libc/W_OK.v.html
./i686-apple-darwin/libc/apple/W_OK.v.html
./i686-apple-darwin/libc/bsd/W_OK.v.html
./i686-apple-darwin/libc/bsd/apple/W_OK.v.html
./i686-apple-darwin/libc/unix/W_OK.v.html
./i686-apple-darwin/libc/unix/apple/W_OK.v.html
./i686-apple-darwin/libc/unix/bsd/W_OK.v.html
./i686-apple-darwin/libc/unix/bsd/apple/W_OK.v.html
./x86_64-unknown-bitrig/libc/netbsdlike/W_OK.v.html
./x86_64-unknown-bitrig/libc/W_OK.v.html
./x86_64-unknown-bitrig/libc/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-bitrig/libc/bsd/W_OK.v.html
./x86_64-unknown-bitrig/libc/unix/netbsdlike/W_OK.v.html
./x86_64-unknown-bitrig/libc/unix/W_OK.v.html
./x86_64-unknown-bitrig/libc/unix/bsd/netbsdlike/W_OK.v.html
./x86_64-unknown-bitrig/libc/unix/bsd/W_OK.v.html
./s390x-unknown-linux-gnu/libc/W_OK.v.html
./s390x-unknown-linux-gnu/libc/notbsd/W_OK.v.html
./s390x-unknown-linux-gnu/libc/unix/W_OK.v.html
./s390x-unknown-linux-gnu/libc/unix/notbsd/W_OK.v.html
./x86_64-apple-darwin/libc/W_OK.v.html
./x86_64-apple-darwin/libc/apple/W_OK.v.html
./x86_64-apple-darwin/libc/bsd/W_OK.v.html
./x86_64-apple-darwin/libc/bsd/apple/W_OK.v.html
./x86_64-apple-darwin/libc/unix/W_OK.v.html
./x86_64-apple-darwin/libc/unix/apple/W_OK.v.html
./x86_64-apple-darwin/libc/unix/bsd/W_OK.v.html
./x86_64-apple-darwin/libc/unix/bsd/apple/W_OK.v.html

If I'm not mistaken, the v.html files are a relatively new feature.

[bluss]

Oh wow, some files are repeated up to 64 times for just one architecture. For example ./libc/__SIZEOF_PTHREAD_CONDATTR_T.v.html

[jonas-schievink]

Maybe #35236 ?

[nrc]

It is somewhat surprising that all those files would add up to 2.3GB - each one is only a redirect so should be a hand full of bytes. Could be possible though. That's the only change I've made, I'm not aware of any others than could have done this.

[DemiMarie]

I suspect that there is an exponential blowup somewhere, probably due to building for the Cartesian product of all possible systems.

Unfortunately, there is no good solution short of dynamically-generated pages.

[brson]

Installed doc size on disk in 1.13 is 248 MB. In 1.12 it was 131 MB.

[ollie27]

I haven't been able to reproduce this and unfortunately the gh-pages has been force pushed to... Building the libc docs for all architectures gives 247 MiB with 366,565 files which is insane but it's not 2.3GB. I'm very curious where that number came from.

Anyway I've submitted a patch (#37773) which fixes some inlining issues and it brings it down to 151 MiB with 90,041 files which is better and I think correct given the current design of rustdoc (nearly 3/4 of the files are redirects).

[alexcrichton]

@ollie27 the 2.3 GB was across a number of platforms (explicit duplication), all listed above in @bluss's comment. I gained the size from du -sh ..

If you check out the libc project and run sh ci/dox.sh you can reproduce the problem. For the compiler rustc 1.13.0 (2c6933acc 2016-11-07) that produces a target/doc directory 1.6G large. The extra space measured was probably git output.

Conversely, for rustc 1.10.0 (cfcb716cf 2016-07-03) the size of the output directory is 349M. (hence the regression)

[ollie27]

du -sh . gives the size in blocks which I don't think is very useful here. I was using the sum of the sizes of the files.

Here's the not particularly scientific results I get:

Rust version	Sum of filesizes	No. of files	Blocks (not that it really matters)
1.10	141 MiB	184781	341M
1.11	188 MiB	183573	849M
1.12	188 MiB	183573	849M
1.13	247 MiB	366584	1.6G
master(ish)	247 MiB	366566	1.7G
master(ish) + #37773	151 MiB	90042	513M

I believe the size increase from 1.10 to 1.11 was caused by #34068, which was a bugfix and so was intentional. From 1.12 to 1.13 was caused by #35236, so again intentional.

I don't think this issue is tracking a regression unless you consider #35236 to be a regression. It did nearly double the number of files rustdoc generates.

The issues with inlining that are fixed by #37773 existed in 1.10 and probably before as well. libc just happens to be a particularly bad case.

[alexcrichton]

Awesome, thanks for the update! If #37773 solves the libc use case them I'm fine with that, libc is sort of an extreme version of the docs.

[brson]

The difference in size between the files and the blocks does seem to matter here. That's 1.6 GB of real disk space that's going unused right? And I'd guess they are the direct result of all the little redirect files.

@ollie27's results look great though.

[brson]

We think this is fixed but need to verify.

[alexcrichton]

Indeed fixed!