Run crates.io inside Travis (without Docker)

[kzys]

This is the first step towards #868.

Personally I still prefer #1814 (Docker version) over this one. We have docker-compose.yml for development, Procfile and buildpacks for production, and this shell script for CI, which might cause "works on my machine" problem.

[rust-highfive]

r? @sgrif

(rust_highfive has picked a reviewer for you, use r? to override)

[kzys]

@jtgeibel / @sgrif - Can we have a S3 bucket to share artifacts between jobs?

https://docs.travis-ci.com/user/build-stages/

It is important to note that jobs do not share storage, as each job runs in a fresh VM or container. If your jobs need to share files (e.g., using build artifacts from the “Test” stage for deployment in the subsequent “Deploy” stage), you need to use an external storage mechanism such as S3 and a remote scp server.

This change currently builds the Rust backend again on Integration Tests job.

[kzys]

Welcome back @carols10cents! Can you take a look this PR, since you are the one who had opened #868?

[carols10cents]

Hi @kzys! Thank you for your patience! I will get to this once I've finished with the PRs that have been open longer than this one has.

[jtgeibel]

I've added some review comments. At a high level, I'd also like to see a more thorough integration test implemented, and a clearer path to coordinating multiple tests. Right now I suspect this test would pass even if we didn't boot the backend server. I think we'll need way to load test fixtures into the database, and verify the HTML contains the expected data.

Basically, I had envisioned that integration tests would probably require us writing some sort of framework that would coordinate all of this, and I'm not sure that a shell script will be sufficient long term. Of course, we'll need to start somewhere and take incremental steps to get to that point, but I'm not sure I see the advantage of merging just this current test at this point and running it on CI for all future PRs. I am however open to leaving this PR open as a place where we can experiment with building out this infrastructure further, I just don't expect that we would be merging this until it is in a state that it will catch potential problems.

I hope I'm not coming off as pessimistic here. I expect that we won't really start to see the advantage of such integration tests until we get Fastboot enabled and then implemented on some of the dynamic endpoints. I'd like to get back into reviewing the Fastboot stuff soon, and I definitely see integration tests providing lots of value in that context.

[jtgeibel]

Even though we have cache.cargo: true above, it looks like this job isn't actually caching any of our Rust build artifacts. I recently realized that travis seems to use different VM images depending on the selected language type. It seems like other behavior, like caching here, must also be dependent on the language type.

This brings me to a general thought. We previously run the frontend tests within our rust: stable job. It seems like it might be cleaner to combine these back into a single job and run any integration tests there as well. That would reduce the duplication of running the split jobs and duplicating the build work from both of those in this integration job. The downside is that I'm starting to see that Travis doesn't really seem to be set up to gracefully handle multiple languages/frameworks within the same job.

[kzys]

I personally prefer the separation. We tend to have either JavaScript-only or Rust-only PRs. The separation helps developers understand what's not working - and which test failures are false-positives.

Between stages, Travis doesn't share any files. Can't we have a S3 bucket to keep intermediate artifacts?

https://docs.travis-ci.com/user/build-stages/#data-persistence-between-stages-and-jobs

[jtgeibel]

I don't have access to create an S3 bucket for intermediate artifacts, but I've added it to the agenda at rust-lang/crates-io-cargo-teams#60 to discuss at the next team meeting.

[kzys]

Another path we could take is deploying to Heroku, as a part of integration testing.

Pros:

• We don't have to worry about artifact sharing.
• It is closer to production and we can even test Nginx parts.
• Looking test failures would be easier, since developers can access the instance directly from their browsers.

Cons:

• We cannot do that per-PR basis. The deployment may be happening only from master.
• We need to have an isolated database instance for testing.

What do you think?

[kzys]

Let me close this PR for now, since I would not work on that in 2019.