Excessive CI compile times

[matthewkmayer]

TravisCI can take upwards of 40 minutes to compile Rusoto on stable. We're also hitting timeouts for beta.

It's high time to look at improving this. I think this is separate from #316 which will make things faster for crate users.

Thoughts:

• make reproducable benchmarks to get hard data
• profile codegen

[matthewkmayer]

I ran some quick numbers in a branch on Rust 1.11.0 and our codegen time is roughly three minutes on my laptop.

From codegen:

for service in services {
        let sw = Stopwatch::start_new();
        generate(service, out_path);
        println!("generation of service took {}ms",  sw.elapsed_ms());
    }

The meat of compile time is probably in the five million lines of code we're shoving into LLVM.

cargo rustc --features "all" -- --emit llvm-ir && wc -l target/debug/rusoto.ll
Compiling rusoto v0.16.0 (file:///Users/matthew/Documents/rusoto)
5102756 target/debug/rusoto.ll

Total compile time from after cargo clean: $ time cargo build --features all:

real    16m10.078s
user    18m2.163s
sys 0m37.334s

[matthewkmayer]

A common theme is our call to serde_generate taking 90%+ of a service's compile time:

generate_client took 13ms
generate_prelude took 0ms
generate_types took 55ms
generate_error_types took 13ms
botocore_generate took 107ms
serde_generate took 4118ms
generation of ecs took 4225ms

service_data capacity: 1046528
generate_client took 53ms
generate_prelude took 0ms
generate_types took 628ms
generate_error_types took 56ms
botocore_generate took 915ms
serde_generate took 11427ms
generation of ec2 took 12342ms

serde-rs/serde#540 may help us in the future.

[adimarco]

I think we run serde_generate on ALL services, not just the JSON ones. That seems like low hanging fruit for optimization, as we only need to run it on JSON services.

[matthewkmayer]

Yes, I just had ec2 and ecs runs handy for numbers.

[dtolnay]

serde-rs/serde#540 may help us in the future.

I tested the following:

$ cargo expand # ...
$ cargo clean -p rusoto_codegen -p rusoto
$ cargo expand # ...

I looked at the time of the last command as a way to measure how long it takes to do just codegen, not counting the time to compile any dependencies.

1. Using syntex (the default): 147 seconds
2. Using serde_macros: 46 seconds
3. Using Macros 1.1 without syntex (from No more syntex for serde_derive serde-rs/serde#548): 49 seconds

The generated code is all the same, which means after codegen all three take the same amount of time.

So based on this, Macros 1.1 will be about as fast as serde_macros is now. The improvement over the current default will be about 100 seconds of codegen + about 60 seconds of not compiling a Syntex stack, which is ~3 minutes saved out of the 10 minutes total (from scratch, on my computer).

[matthewkmayer]

Something else we can do is not generate deserializers for send-only data types and not generate serializers for receive-only data types in codegen. I've done that in #375 for ec2 but we should do the same for other codegen implementations.

[matthewkmayer]

Macros 1.1 probably won't hit stable until 2017 or so.

On the bright side, some compiler performance regressions have been fixed so Rust 1.12.0, shipping tomorrow, will reduce our compile time from 37 minutes to 27 minutes per TravisCI build at Rusoto v0.17.0. Almost back to 1.9.0 speeds.

[matthewkmayer]

My understanding of how we have TravisCI running our builds on stable is off: compile time improvements on stable have only occurred from the code trimming work I've done and we're not seeing better times with Rust 1.12.0.

Current thought is our TravisCI config for running cargo doc via travis-cargo is adding a significant amount of time to the build.

(Edit)

time cargo doc --features all takes over seven minutes on my laptop, that's probably where our extra build time is spent.

[indiv0]

We probably want to double check we're only generating documentation for rusoto and codegen, and not any external libraries. Other than that I don't think we can speed up documentation generation outside of separating it back out of Travis CI (which I'd rather not do).

[matthewkmayer]

We're generating docs for everything not just our code. Just didn't get to looking hard at that yet.

[matthewkmayer]

I'm going to give caching cargo deps a try in a branch.

More details: travis-ci/travis-ci#4470 .

[matthewkmayer]

Appveyor compile times about doubled with the move from Rust 1.11 to 1.12:

The PR with a README change: 27 minutes on Rust 1.11.

The master branch after just that merge: 53 minutes on Rust 1.12.

Our appveyor config doesn't do anything special for PRs or branches so it should be apples to apples except the Rust version bump.

[matthewkmayer]

I think we've reached what we can do before Macros 1.1 lands.

Another good option is to shatter the crate. We'd make a new crate for each service. Cargo has a workspace capacity that's been added since we last looked at shattering the crate. I believe making a crate for each service allows parallel compiling, which we don't have. It'll also prevent us from stuffing so many lines into LLVM for compilation.

Any objects to closing this?

[matthewkmayer]

Adding S3 as a generated service is causing timeouts on TravisCI's OSX builds in #321 . We may want to reopen this to see what other improvements can be made. I say we wait to see how the next release of Rust changes compilation times, we could punt this a little longer.