Add GUI tests

[GuillaumeGomez]

Part of #1494.

I finally took the time to wrap something together. So in this case, I took what we do for rustdoc and simply ported it to docs.rs. I added 2 GUI tests so you can take a look at what they look like. The documentation for the framework is here.

Do you think this is the right approach or would you prefer another framework?

[GuillaumeGomez]

I'm a bit stuck with the current docker situation. So to sum up what I tried to and what my issue is:

• I want to have a docs.rs environment set up with a "clean" database with only the crates/updates we need for the GUI tests. The other point is that it doesn't "pollute" the existing DB by adding new crates.
• It should be reproducible easily.
• The docker image doesn't need to be rebuilt every time you make a GUI test update.

So now the problem is that I think putting everything in one docker container isn't a great idea because it makes a big image which needs to be rebuilt every time there is a change on docs.rs source code.

The other problem is that apparently, we can't start docs.rs web server in the container: it exists right away (without an error).

So first: is it the right approach? Secondly: any help would be very appreciated.

[GuillaumeGomez]

So I was finally able to make it work like I wanted. I even added it to the docker-compose and I added a shell script to make it simpler to run.

[GuillaumeGomez]

Maybe ping @Nemo157 ?

[GuillaumeGomez]

@syphar It's "ready" now, we need to discuss about if this looks good to you or if it seems to complicated to maintain and eventually try to pick another solution.

[syphar]

@syphar It's "ready" now, we need to discuss about if this looks good to you or if it seems to complicated to maintain and eventually try to pick another solution.

Thanks for the ping!
This topic was next on my list to dig into.

For my understanding: rustdoc is actively using the same framework?

[GuillaumeGomez]

Yes. You can see it being used in src/test/rustdoc-gui.

[GuillaumeGomez]

I updated to the last version of the framework and added some documentation in case you want to run it locally. So with this, it should be ready.

From here, the plan is then to test as much as possible our UI, especially the top-navbar and all docs.rs pages.

[GuillaumeGomez]

CI finally passed! \o/

PR is now ready. :)

[syphar]

only one last comment from my side. I wonder if we can safe time on the setup, or if caching will help a a little, 20 minutes, are quite long.

I would like to have a short test from @Nemo157 because I can't currently run the tests on my mac

[GuillaumeGomez]

only one last comment from my side. I wonder if we can safe time on the setup, or if caching will help a a little, 20 minutes, are quite long.

Yes, 20 minutes is way too long. If you know how to cache it, I'd be very happy to do it.

I would like to have a short test from @Nemo157 because I can't currently run the tests on my mac

Is it because of this PR or something specific to your mac?

[syphar]

only one last comment from my side. I wonder if we can safe time on the setup, or if caching will help a a little, 20 minutes, are quite long.

Yes, 20 minutes is way too long. If you know how to cache it, I'd be very happy to do it.

do you know where most of the time is spent?

I would like to have a short test from @Nemo157 because I can't currently run the tests on my mac

Is it because of this PR or something specific to your mac?

I mean these GUI tests don't run, the other tests do.

the -- build crate ... things don't run, and I have to use docker compose run web build crate...

but running the test script leads to another error related to the host / container platform (I'm on an M1).

But IMO this doesn't block this PR, I'll figure it out afterwards.

[GuillaumeGomez]

Oh wow. It took only 8 minutes without docker build.

[Nemo157]

the -- build crate ... things don't run, and I have to use docker compose run web build crate...

I also can't use a local build crate, the rustup that rustwide downloads doesn't like nixos.

[GuillaumeGomez]

Wow, weird. I thought using docker would prevent all these issues. ^^'

[Nemo157]

rustwide does all the toolchain installation outside docker containers for some reason (it should be possible to move that all into the containers, which would probably make it work on macos just fine too).

[GuillaumeGomez]

I see. Do you want it to be fixed before we merge this PR or after?

[Nemo157]

It's orthogonal to this PR (and given maintenance status of rustwide unlikely to happen soon). I plan to see if I can adapt the script here to run everything inside docker-compose containers, can probably get around to it today if it's not too complicated.

[GuillaumeGomez]

Thanks! :D

[Nemo157]

I got it working in 16d4957 and managed to run them locally, but I think this can be much simpler after the refactoring in #2269, so I'd merge as is and I'll try to finish that up and update the script later.

[GuillaumeGomez]

I'll let you handle the merge when you're ready then. Thanks for the help!

[syphar]

@Nemo157 so you think we should merge this PR now, and then improve on #2269 ?

[Nemo157]

Yep

[GuillaumeGomez]

Awesome! \o/