Gl-CI integration for master

[LynxAbraxas]

In regard to #68, this PR integrates the configuration for GL-CI from https://github.com/LynxAbraxas/ctp2DF/ into https://github.com/civctp2/civctp2 to enable automated GUI-smoke tests for new commits (possibly only for master).
It is adjusted to avoid git submodules and use GL-CI artifacts for providing ctp2CD/ instead.

[LynxAbraxas]

OK, jobs succeed on travis and on GL. Took me half a day to get the various incantations right.

@MartinGuehmann If you consider accepting this PR, you will need to create an account on GL for the user civctp2.
Then create https://gitlab.com/civctp2/civctp2/ as a private project to ensure that the final docker images are not public. Populate from GH before accepting this PR.
Clone my ctp2CD/ to https://gitlab.com/civctp2/ctp2CD/ and keep that private as well.
The .gitlab-ci.yml of ctp2CD/ will create an artifact of the CD files, which will be downloaded by .gitlab-ci.yml of civctp2/ (to avoid ctp2CD/ to be a git submodule). For that you need to create an API token (civctp2-user settings) and pass that as a CI environment variable named ARTIFACTKEY (civctp2-project settings).
Then set up the civctp2-project to mirror (https://gitlab.com/civctp2/civctp2/settings/repository) GH with pull and Trigger pipelines for mirror updates
When you then merge this PR in master on GH, GL should catch that on the next mirror execution and then will start to run the CI-pipeline for the first time. This will take around 1-2h for the first time because the docker images for caching are not there yet. For the following pipelines, the former created docker images will be pulled from the GL registry and used for caching, reducing the run time considerably (~20min).

This shall be my last ctp2 contribution for at least this year...

[MartinGuehmann]

@MartinGuehmann If you consider accepting this PR, you will need to create an account on GL for the user civctp2.

I created the user account civctp2.

Then create https://gitlab.com/civctp2/civctp2/ as a private project to ensure that the final docker images are not public. Populate from GH before accepting this PR.

I haven't created the project, yet. I can import something from an existing project. I just don't know what populate from GitHub mean. Is it important stuff from https://github.com/civctp2/civctp2/? And if yes, what to import?

Clone my ctp2CD/ to https://gitlab.com/civctp2/ctp2CD/ and keep that private as well.

I think this is clear.

The .gitlab-ci.yml of ctp2CD/ will create an artifact of the CD files, which will be downloaded by .gitlab-ci.yml of civctp2/ (to avoid ctp2CD/ to be a git submodule). For that you need to create an API token (civctp2-user settings) and pass that as a CI environment variable named ARTIFACTKEY (civctp2-project settings).

I think, I have seen that option on GitHub.

Then set up the civctp2-project to mirror (https://gitlab.com/civctp2/civctp2/settings/repository) GH with pull and Trigger pipelines for mirror updates

Here, I have to see what this is, but I think I will ask when I am there.

When you then merge this PR in master on GH, GL should catch that on the next mirror execution and then will start to run the CI-pipeline for the first time. This will take around 1-2h for the first time because the docker images for caching are not there yet. For the following pipelines, the former created docker images will be pulled from the GL registry and used for caching, reducing the run time considerably (~20min).

So when I merge I don't have to worry about this anymore, I guess.

This shall be my last ctp2 contribution for at least this year...

That's a pitty @LynxAbraxas, but well I consider the feedback about this as part of the contribution.

[LynxAbraxas]

I created the user account civctp2.

That's great!

I haven't created the project, yet. I can import something from an existing project. I just don't know what populate from GitHub mean. Is it important stuff from https://github.com/civctp2/civctp2/? And if yes, what to import?

You can either use the import function in the GL UI, or you can just create an empty civctp2 project and push your local civctp2 clone to GL like:

git remote add GL https://gitlab.com/civctp2/civctp2 # add GL as remote
git fetch upstream # sync with upstream, i.e. GH
git fetch upstream --tags # just to be sure
git push GL --all --tags # push all branches and tags to GL, best check in GL UI if tags are there

Same for ctp2CD/.

I think, I have seen that option on GitHub.

Not on GH but GL because the artifact is downloaded in the GL-CI environment.
Should be under "Environment variables" in https://gitlab.com/civctp2/civctp2/settings/ci_cd.

That's a pitty @LynxAbraxas, but well I consider the feedback about this as part of the contribution.

I will try to give feedback, just don't expect it regularly nor soon.

[MartinGuehmann]

I will try to give feedback, just don't expect it regularly nor soon.

I would say as long as we get it done, eventually: This must be good enough. :D

[MartinGuehmann]

I think that I set it up except, that it did not download the CD. But that could be because, I followed your instructions in the order of appearance. I guess I had to create ARTIFACTKEY before I had cloned ctp2CD.

[LynxAbraxas]

Did GL create an artifact for ctp2CD after you cloned that?
ARTIFACTKEY needs to be a civctp2-project setting.
If the artifact for ctp2CD was created, you could try running a new pipeline manually by clicking "Run Pipeline" under https://gitlab.com/civctp2/civctp2/pipelines

@MartinGuehmann can you invite me under GitLab to the civctp2 repos as a reporter or above, so I can take a look?

[MartinGuehmann]

Did GL create an artifact for ctp2CD after you cloned that?
ARTIFACTKEY needs to be a civctp2-project setting.
If the artifact for ctp2CD was created, you could try running a new pipeline manually by clicking "Run Pipeline" under https://gitlab.com/civctp2/civctp2/pipelines

@MartinGuehmann can you invite me under GitLab to the civctp2 repos as a reporter or above, so I can take a look?

I don't think it created the artifact. Anyway, I invited you as reporter. I would also you give you access above, but since you don't wanna be sucked in more, I went for reporter.

I also noticed that the civctp2 project was created public, I should have fixed that. The ctp2CD project was created private as it should be.

[LynxAbraxas]

Thanks @MartinGuehmann. So far I can only see civctp2, so I could not check if an artifact was created for ctp2CD. Did you invite me as a reporter also for ctp2CD?
I think the problem is that this PR is not yet part of the repos when it is normally cloned, and without there is no .gitlab-ci.yml that would initiate the GL-CI. So perhaps try this (https://help.github.com/articles/checking-out-pull-requests-locally/#modifying-an-inactive-pull-request-locally):

git fetch upstream pull/84/head:GL-CI4master
git remote add GL https://gitlab.com/civctp2/civctp2/ # if you have not added the GitLab repos already as a remote
git push GL GL-CI4master

and then check, if there is a pipeline started for GL-CI4master.

[MartinGuehmann]

Thanks @MartinGuehmann. So far I can only see civctp2, so I could not check if an artifact was created for ctp2CD. Did you invite me as a reporter also for ctp2CD?
I think the problem is that this PR is not yet part of the repos when it is normally cloned, and without there is no .gitlab-ci.yml that would initiate the GL-CI. So perhaps try this (https://help.github.com/articles/checking-out-pull-requests-locally/#modifying-an-inactive-pull-request-locally):

Actually, I found the problem, since it is a bit confusing, I cloned https://github.com/LynxAbraxas/ctp2DF/, as a private project.

The other thing is as far as I can see I can only add projects to the civctp2 account if I login as civctp2. Since, you did not grant access to civctp2, I cannot see the private projects. Well, as MartinGuehmann, I could get the repository file from your GitLab account. I used the URL for importing, which is what it is doing right now. Actually, I expected a password and a username since I imported it as civctp2.

Well, let's see if it is just that.

[LynxAbraxas]

I can see that GL picked up the merge of the last PR. @MartinGuehmann were you now able to clone ctp2CD? Could you give me reporter status for that as well?

[MartinGuehmann]

I can see that GL picked up the merge of the last PR. @MartinGuehmann were you now able to clone ctp2CD? Could you give me reporter status for that as well?

I could create the project, and give you reporter status. But so far as I can see it fails to import it, I gave the credentials of civctp2 as the manual says. So right now it is important, we will see whether it did import it.

[LynxAbraxas]

Thanks @MartinGuehmann, I can see the ctp2CD project now, but it does not contain the repository, so no artifact gets created. Perhaps try it manually with:

git clone https://MartinGuehmann@gitlab.com/LynxAbraxas/ctp2CD
cd ctp2CD
git remote add upstream https://gitlab.com/civctp2/ctp2CD
git push upstream --all --tag

[MartinGuehmann]

Thanks @MartinGuehmann, I can see the ctp2CD project now, but it does not contain the repository, so no artifact gets created. Perhaps try it manually with:

git clone https://MartinGuehmann@gitlab.com/LynxAbraxas/ctp2CD
cd ctp2CD
git remote add upstream https://gitlab.com/civctp2/ctp2CD
git push upstream --all --tag

So I used those commands manually. It didn't like the combination of the --all and --tag flag, so I left out --tag. And I had to set a password on GitLab, so far I logged in via GitHub. It didn't copy that automatically.

[LynxAbraxas]

That worked well, the ctp2CD artifact is there:
https://gitlab.com/civctp2/ctp2CD/-/jobs/163464613/artifacts/browse
You can get there by clicking through starting from the pipelines:
https://gitlab.com/civctp2/ctp2CD/pipelines

So now only the commits of this PR are missing in https://gitlab.com/civctp2/civctp2/
Just do

git fetch upstream pull/84/head:GL-CI4master
git remote add GL https://gitlab.com/civctp2/civctp2/ # if you have not added the GitLab repos already as a remote
git push GL GL-CI4master

and you should see a pipeline started under CI-CD:
https://gitlab.com/civctp2/civctp2/pipelines

[LynxAbraxas]

However I also get:
Home directory not accessible: Permission denied

Not essential, but can be mended with: 7ef24e9

In regard to #94 (comment)
I also tested upgrading the dockerimage base to ubuntu:18.04 (bionic) which surprisingly only needed an additional minor change to work, so added that commit from ctp2DF here as well.

When you merge this PR, the pipeline should run again and populate the DIs for the master branch under: https://gitlab.com/civctp2/civctp2/container_registry
And you should see the screen-shots from the automated tested under https://gitlab.com/civctp2/civctp2/blob/master/GL-CI.md

If you have docker set up on your local machine, you should then be able to run the DIs locally with e.g.:
./run.sh -v ~/ctp2OGGs/:/opt/ctp2/ctp2_program/ctp/music/:ro -v ~/ctp2videos/:/opt/ctp2/ctp2_data/default/videos/:ro registry.gitlab.com/civctp2/civctp2/master:latest ./ctp2

Which expects the OGGs in ~/ctp2OGGs/ and the videos converted as described here:

civctp2/doc/README.linux

Lines 254 to 263 in c2dafb8

	10.Movies
	ffmpeg and sdl_ffmpeg > 0.7.0 are used for movie playing now. Strangely ffmpeg can't
	play the movies of the CD. So you'll have to convert them to a format ffmpeg
	can handle.
	One way of doing that is with mencoder (you can choose any codec ffmpeg
	can play, even h264;):
	shell> cd /home/foo/CTP2/ctp2_program/ctp
	shell> for i in `ls /media/cdrom/Setup/data/Max/ctp2_data/default/videos/`; do mencoder -ovc lavc -lavcopts vcodec=mpeg4 -srate 44100 -af resample=44100 -oac lavc /media/cdrom/Setup/data/Max/ctp2_data/default/videos/$i -o $i; done

If the DI does not exist locally docker will pull the image from the GL registry, for that you need to be logged in with docker login -u civctp2 registry.gitlab.com. If the DI exists locally, docker will not check whether a newer DI exists when useing docker run instead used docker pull registry.gitlab.com/civctp2/civctp2/master:latest for that. docker images lists your local images.

As it basically seem to work now, I'd say the PR is ready to go in.

[MartinGuehmann]

As it basically seem to work now, I'd say the PR is ready to go in.

Then, let's merge.

[LynxAbraxas]

Pipeline for master started and nearly finished, only play-game_build-city test failed:
https://gitlab.com/civctp2/civctp2/pipelines/48986820
play-game_build-city test is the most sensitive (error prone) test so sometimes the 3 tries configured by default do not suffice. @MartinGuehmann do you mind hitting "retry" for the pipeline once or twice to see if that suffices?

[MartinGuehmann]

Once was enough. Is there an error log to see what it is?

[LynxAbraxas]

Thanks @MartinGuehmann, seems to work as expected now. See e.g.
https://gitlab.com/civctp2/civctp2/commits/master
where also f8f1c59 is listed as fully tested and there is a DI for that commit in the registry: https://gitlab.com/civctp2/civctp2/container_registry.

[MartinGuehmann]

Thanks @MartinGuehmann, seems to work as expected now. See e.g.
https://gitlab.com/civctp2/civctp2/commits/master
where also f8f1c59 is listed as fully tested and there is a DI for that commit in the registry: https://gitlab.com/civctp2/civctp2/container_registry.

Should be f8f1c59. (Interesting, the preview eats the final 4.) As far as I can see, it only test master, but not any pull request branches. The GitLab project seems only to mirror master and not any branches or pull requests, either, unless they get merged.

[LynxAbraxas]

Once was enough. Is there an error log to see what it is?

Yes, each job of a pipeline has its own log (and artifacts) like
https://gitlab.com/civctp2/civctp2/-/jobs/166709948. You can get there by clicking through the GL UI starting from the pipelines -> click on status button (https://gitlab.com/civctp2/civctp2/pipelines/48986820) -> job button for play-game_build-city (https://gitlab.com/civctp2/civctp2/-/jobs/166731243) -> then on the lower right side you see the full list of jobs for that pipeline where also the failed ones are listed (in this case 3 for play-game_build-city), see also #105 and #104.

[LynxAbraxas]

@MartinGuehmann were you able to run ctp2 from the GL DI locally on your PC, as described above in #84 (comment)?

[LynxAbraxas]

@MartinGuehmann you can see the CI minutes used so far for the month under https://gitlab.com/profile/pipeline_quota if logged in as civctp2.
If 2000 per month are not enough there is the possibility (apart from upgrading to a paid plan) to set up a runner on a local machine that can then be used instead of the shared-runners from GL. I used that for testing the multi-threaded build because the GL-CI shared-runners are not guaranteed to offer more than one CPU.

[MartinGuehmann]

@MartinGuehmann you can see the CI minutes used so far for the month under https://gitlab.com/profile/pipeline_quota if logged in as civctp2.
If 2000 per month are not enough there is the possibility (apart from upgrading to a paid plan) to set up a runner on a local machine that can then be used instead of the shared-runners from GL. I used that for testing the multi-threaded build because the GL-CI shared-runners are not guaranteed to offer more than one CPU.

Well it says, unlimited minutes and that the project does not use any shared runners. Which is I guess why that is unlimited.

I haven't tried to run the docker image locally, I guess the advantage is that it does not consume server minutes. And that's why it just tests the merges, than the pull requests, too.

[LynxAbraxas]

Well it says, unlimited minutes and that the project does not use any shared runners.

Odd, top left of
https://gitlab.com/civctp2/civctp2/-/jobs/167301425
says "Runner: shared-runners-manager-6.gitlab.com (#380987)" so it is definitly using shared-runners. Perhaps you need to log out and in again?

I haven't tried to run the docker image locally, I guess the advantage is that it does not consume server minutes.

Running the DI locally does not correspond to running the GL-CI pipeline and its jobs! You would not have a DI from GL without the GL-CI pipeline (you would have to run docker build from the build job first). Running the DI locally is for testing the readily built cpt2 more extensivily or specificly than the automatic test do, e.g. for /rnd or /wd or the video playing.

And that's why it just tests the merges, than the pull requests, too.

Well, GL mirrors GH and that only includes branches and tags that you get with git fetch or git clone but that does not include PR and their branches, so GL does not see the commits of a PR until they are merged into the main repos. You can force testing a PR similar to #84 (comment):

git fetch upstream pull/110/head:SDL_ffmpegUpdate4GL-CI_PR110
# git remote add GL https://gitlab.com/civctp2/civctp2/ # if you have not added the GitLab repos already as a remote
git push GL SDL_ffmpegUpdate4GL-CI_PR110

[LynxAbraxas]

@MartinGuehmann be aware that the GL-CI tests are language sensitive! I just stumbled into this trap, i.e. if you ever change https://gitlab.com/civctp2/ctp2CD to use another language than English some tests of civctp2 will fail and some not. The GL-CI configuration of civctp2 is such that it uses the latest artifact from ctp2CD.

[MartinGuehmann]

@MartinGuehmann you can see the CI minutes used so far for the month under https://gitlab.com/profile/pipeline_quota if logged in as civctp2.

I must read the damn thing, we have used up 7%, which I find already quite a lot. But that should be manageable.

Well, GL mirrors GH and that only includes branches and tags that you get with git fetch or git clone but that does not include PR and their branches, so GL does not see the commits of a PR until they are merged into the main repos. You can force testing a PR similar to #84 (comment):

With the quota in mind, I would restrict it to merge commits and won't use it for the other branches. I assume ones we have exhausted the quota it won't just run anymore.

@MartinGuehmann be aware that the GL-CI tests are language sensitive! I just stumbled into this trap, i.e. if you ever change https://gitlab.com/civctp2/ctp2CD to use another language than English some tests of civctp2 will fail and some not. The GL-CI configuration of civctp2 is such that it uses the latest artifact from ctp2CD.

What tests are that in particular, as far as I can see we have now the full set of text files and ldl files (except for Japanese), for English and German we also have the sound files. So I would expect that only playing sound would fail. And right now I am not aware that it is testing sound, of course I could be wrong.

[LynxAbraxas]

we have used up 7%, which I find already quite a lot.

Further runs should use less due to the caching from previous docker images in the build job, though compilation time is still quite significant, but e.g. pulling packages for the base system will come from the cache as long as there is no newer base image.
For example, if only the tests are changed, the build job will fully be used from the caches because docker ignores changes to tests/:
https://github.com/civctp2/civctp2/blob/f8f1c594c4edd1b10067b40d009212e12d343605/.dockerignore

I assume ones we have exhausted the quota it won't just run anymore.

Yes, until the 1st of the next month when the 2000 are refreshed.
Should you be in very urgent need of more, you could also test with your GL-MartinGuehmann account.

What tests are that in particular

Those where the button text changes with the language;-) like Cancel <-> Abbrechen. It also depends on how strict Sikuli is configured to match the reference image. For example the start-game test succeeds if the the start screen (https://github.com/civctp2/civctp2/blob/f8f1c594c4edd1b10067b40d009212e12d343605/tests/ctp2start-scr.png) is visible

civctp2/tests/start-game.sikuli/start-game.py

Line 19 in f8f1c59

f.find("ctp2start-scr.png")

.
The overall image is quite large so the actual text and the compile date in the ref-image does not matter.
In play-game_build-city, there are quite a view small and similar buttons in the build manager, so sikuli needs to be told to use a high similarity for finding the pattern (

civctp2/tests/play-game_build-city.sikuli/play-game_build-city.py

Line 25 in f8f1c59

click(Pattern("ctp2city-build_close-btn.png").similar(0.90))

) as it otherwise clicks on other similar buttons that it finds first, therefore play-game_build-city test is so far the most sensitive test.

And right now I am not aware that it is testing sound

No sound test so far, was intending to do so with #104.

[MartinGuehmann]

Further runs should use less due to the caching from previous docker images in the build job, though compilation time is still quite significant, but e.g. pulling packages for the base system will come from the cache as long as there is no newer base image.
For example, if only the tests are changed, the build job will fully be used from the caches because docker ignores changes to tests/:
https://github.com/civctp2/civctp2/blob/f8f1c594c4edd1b10067b40d009212e12d343605/.dockerignore

I assume ones we have exhausted the quota it won't just run anymore.

Yes, until the 1st of the next month when the 2000 are refreshed.
Should you be in very urgent need of more, you could also test with your GL-MartinGuehmann account.

Well, then I will just go on as I see fit and then see.

What tests are that in particular

Those where the button text changes with the language;-) like Cancel <-> Abbrechen. It also depends on how strict Sikuli is configured to match the reference image. For example the start-game test succeeds if the the start screen (https://github.com/civctp2/civctp2/blob/f8f1c594c4edd1b10067b40d009212e12d343605/tests/ctp2start-scr.png) is visible
civctp2/tests/start-game.sikuli/start-game.py

Line 19 in f8f1c59

f.find("ctp2start-scr.png")
.
The overall image is quite large so the actual text and the compile date in the ref-image does not matter.
In play-game_build-city, there are quite a view small and similar buttons in the build manager, so sikuli needs to be told to use a high similarity for finding the pattern (
civctp2/tests/play-game_build-city.sikuli/play-game_build-city.py

Line 25 in f8f1c59

click(Pattern("ctp2city-build_close-btn.png").similar(0.90))
) as it otherwise clicks on other similar buttons that it finds first, therefore play-game_build-city test is so far the most sensitive test.

That explains a lot, and you can't just click at a certain position of the screen, since these buttons always appear at the same spot of the screen, at least after a restart without moving the windows.

[LynxAbraxas]

Just noticed with https://gitlab.com/civctp2/civctp2/pipelines/charts that the pipeline for f8f1c59 took longer than that for e0edb94. Taking a look at the job logs (https://gitlab.com/civctp2/civctp2/-/jobs/167301425/raw and https://gitlab.com/civctp2/civctp2/-/jobs/170573090) shows that the pipeline for f8f1c59 pulled in a newer base image of ubuntu (as expected with --pull in

civctp2/.gitlab-ci.yml

Line 26 in e0edb94

- docker build --pull

) which causes invalidation of the former cache and all follwoing docker instructions to be re-executed (as expected).
@MartinGuehmann for that case it is good to have a monthly schedule, just add one at https://gitlab.com/civctp2/civctp2/pipeline_schedules. That will then run a pipeline that tests if the whole docker config is still compatible with the newest ubuntu:18.04 DI and following pipelines will have a valid cache regarding the new base.