Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change: [Actions] stop relying on external actions for trivial stuff, and rely on cmake tools to build, test and pack #8718

Merged
merged 5 commits into from Feb 24, 2021

Conversation

@glx22
Copy link
Contributor

@glx22 glx22 commented Feb 21, 2021

Motivation / Problem

We used run-vcpkg to manage vcpkg for windows and macos targets and run-cmake to build windows targets. But doing the same without these actions is quite trivial, and easier to understand in the workflow.
Also we used to run make to build non windows targets, but it's better to let cmake handle the calls to native build tools.

Description

Implemented the above changes.

Limitations

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

  • The bug fix is important enough to be backported? (label: 'backport requested')
  • This PR affects the save game format? (label 'savegame upgrade')
  • This PR affects the GS/AI API? (label 'needs review: Script API')
    • ai_changelog.hpp, gs_changelog.hpp need updating.
    • The compatibility wrappers (compat_*.nut) need updating.
  • This PR affects the NewGRF API? (label 'needs review: NewGRF')
Copy link
Member

@TrueBrain TrueBrain left a comment

That looks very nice :D

Often I mentioned something on a single place, but I am sure you understand I mean all instances of similar change ;)

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-

This comment has been minimized.

@TrueBrain

TrueBrain Feb 22, 2021
Member

weird restore-key, ending with a -?

This comment has been minimized.

@glx22

glx22 Feb 22, 2021
Author Contributor

It matches all keys if key is a miss, and use the most recent.

This comment has been minimized.

@orudge

orudge Feb 23, 2021
Contributor

I submitted a PR to run-vcpkg (lukka/run-vcpkg#75) which alters the caching there so that it uses the env.ImageVersion variable in the cache key, so that the cache is invalidated if the VM environment is changed. It would be worth us using this in our key (it did cause a problem on macOS before).

(I was waiting for the new release of run-vcpkg, which did happen but I then forgot about it, so never updated OpenTTD...)

This comment has been minimized.

@glx22

glx22 Feb 23, 2021
Author Contributor

Do you remember the exact issue ?
I cache less than run-vcpkg (only vcpkg/installed folder) and I assume vcpkg exe provided in the image works. If vcpkg in image is upgraded, it should detect if installed libs need an upgrade (at least my local vcpkg does that)

This comment has been minimized.

@orudge

orudge Feb 23, 2021
Contributor

I suspect the build logs will have expired now, but I think the error was relating to missing dependencies in libstdc++.dylib or something along those lines, due to a compiler change in the VM template.

I had pinned vcpkg to a particular commit for macOS, but I think the version in the image is new enough to support our arm64 workflow, so it should hopefully be fine.

This comment has been minimized.

@glx22

glx22 Feb 23, 2021
Author Contributor

Well release workflow seems to work https://github.com/glx22/OpenTTD/runs/1946699269?check_suite_focus=true
And it seems run-vcpkg always rebuilt vcpkg executable (even for windows)

This comment has been minimized.

@TrueBrain

TrueBrain Feb 24, 2021
Member

A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.

${{ runner.os } is just macOS for example. Not macos-latest or macos-10.15. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.

It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?

What do you think @glx22 ?

This comment has been minimized.

@glx22

glx22 Feb 24, 2021
Author Contributor

Hmm yes I can replace ${{ runner.os } with ${{ env.ImageOS }}-${{ env.ImageVersion }}.
That means cache is always rebuilt when a new image is used. It was partially the case when using the commit id in the key as it would use cached libs and upgrade them if needed, assuming binary compatibility.
Will be safer to always fully rebuild libs unconditionally.
I think I can remove commit id from the key, and also restore-keys.

.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Show resolved Hide resolved
Copy link
Contributor Author

@glx22 glx22 left a comment

Will address other comments

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-

This comment has been minimized.

@glx22

glx22 Feb 22, 2021
Author Contributor

It matches all keys if key is a miss, and use the most recent.

.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
@glx22 glx22 force-pushed the glx22:cmake_ci branch from 7d70be8 to 48ceb48 Feb 22, 2021
Copy link
Member

@LordAro LordAro left a comment

No issues in principle

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-

This comment has been minimized.

@TrueBrain

TrueBrain Feb 24, 2021
Member

A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.

${{ runner.os } is just macOS for example. Not macos-latest or macos-10.15. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.

It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?

What do you think @glx22 ?

@glx22 glx22 force-pushed the glx22:cmake_ci branch 5 times, most recently from 56cd3bb to 7ac062e Feb 24, 2021
@glx22 glx22 force-pushed the glx22:cmake_ci branch from 7ac062e to 5d90a96 Feb 24, 2021
Copy link
Member

@TrueBrain TrueBrain left a comment

Seriously nice work :)

@glx22 glx22 merged commit 70e4845 into OpenTTD:master Feb 24, 2021
11 checks passed
11 checks passed
Emscripten
Details
Commit checker
Details
Check preview needs update Check preview needs update
Details
Linux (clang, clang++, libsdl2-dev)
Details
Linux (gcc, g++, libsdl2-dev)
Details
Linux (gcc, g++, libsdl1.2-dev)
Details
Mac OS (x64, x86_64)
Details
Windows (windows-latest, x86)
Details
Windows (windows-latest, x64)
Details
Windows (windows-2016, x86)
Details
Windows (windows-2016, x64)
Details
@glx22 glx22 deleted the glx22:cmake_ci branch Feb 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants