Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

azure: drop powershell #5141

Merged
merged 11 commits into from
Jul 21, 2019
Merged

azure: drop powershell #5141

merged 11 commits into from
Jul 21, 2019

Conversation

pks-t
Copy link
Member

@pks-t pks-t commented Jun 24, 2019

Quick and dirty PR to drop all our Powershell scripts and use Bash across all platforms. I bet it won't succeed yet, but I've been a bit lazy to fire up a Windows host :P

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch from 2d3584b to 5777ebf Compare June 24, 2019 16:22
@ethomson
Copy link
Member

Oh, jeez. I think that the MSVC builds are failing because they're running inside Git for Windows (aka msys aka mingw) and so it thinks that you want to build them with mingw.

And I think that the mingw builds are failing because they don't understand Windows-style paths (eg C:\foo) and you need to convert them to /c/foo. There's an app for this that ships with cygwin but ughhhh.

@pks-t
Copy link
Member Author

pks-t commented Jun 25, 2019 via email

@ethomson
Copy link
Member

Should've marked this PR as WIP.

You were clear in setting expectations. I just wanted to 👀 to see how bad it was. 😀

I'm heads down on a bunch of things right now but I'm happy to help look at these when I have some time. (ETA for that unknown.)

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch 10 times, most recently from d5794f7 to 2e3c875 Compare June 27, 2019 12:44
@pks-t
Copy link
Member Author

pks-t commented Jun 27, 2019

Cherry-picked fixes from #5143 as this PR here implicitly enables Werror.

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch 3 times, most recently from 385d53a to e9ab556 Compare June 27, 2019 13:31
@pks-t
Copy link
Member Author

pks-t commented Jun 27, 2019

Jeez, this is quickly ballooning in size due to the ENABLE_WERROR stuff. @ethomson, please feel free to either merge this PR in favour of yours or to cherry-pick any fixes.

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch 6 times, most recently from d4f8cbe to 1689a39 Compare June 28, 2019 10:11
@pks-t
Copy link
Member Author

pks-t commented Jun 28, 2019

/rebuild

@libgit2-azure-pipelines
Copy link

Okay, @pks-t, I started to rebuild this pull request as build #2268.

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch 3 times, most recently from f805f88 to 8620a50 Compare June 28, 2019 12:12
@pks-t pks-t force-pushed the pks/azure-drop-powershell branch 5 times, most recently from 937f2c8 to c1b0edf Compare July 19, 2019 10:48
@pks-t
Copy link
Member Author

pks-t commented Jul 19, 2019

@ethomson: finally got this solved. The fix was to not set PATH at all for our jobs, but instead to only adjust PATH for the CMake invocations.

ethomson
ethomson approved these changes Jul 20, 2019
View reviewed changes
Copy link
Member

@ethomson ethomson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'm approving but unfortunately I merged the other changes first, so there are conflicts.

@pks-t pks-t force-pushed the pks/azure-drop-powershell branch from c1b0edf to 9384a7c Compare July 20, 2019 16:36
@pks-t
Copy link
Member Author

pks-t commented Jul 20, 2019

Rebased to fix conflicts

pks-t added 11 commits July 20, 2019 19:10
@pks-t
fuzzers: provide test targets
86ecd60 
Instead of having to find the fuzzer executables in our Azure test
scripts, provide test targets for each of our fuzzers that will
run them with the correct paths.
@pks-t
clar: provide ability to set summary file via environment
fe3b5da 
As different test suites for our CI are mostly defined via CMake, it's
hard to run those tests with a summary file path as that'd require us to
add another parameter to all unit tests. As we do not want to
unconditionally run unit tests with a summary file, we would have to add
another CMake build parameter for test execution, which is ugly.

Instead, implement a way to provide a summary file path via the
environment.
@pks-t
tests: execute leak checker via CTest directly
d827b11 
Right now, we have an awful hack in our test CI setup that extracts the
test command from CTest's output and then prepends the leak checker.
This is dependent on non-machine-parseable output from CMake and also
breaks on various ocassions, like for example when we have spaces in the
current path or when the path contains backslashes. Both conditions may
easily be triggered on Win32 systems, and in fact they do break our
Azure Pipelines builds.

Remove the awful hack in favour of a new CMake build option
"USE_LEAK_CHECKER". If specifying e.g. "-DUSE_LEAK_CHECKER=valgrind",
then we will set up all tests to be run under valgrind. Like this, we
can again simply execute ctest without needing to rely on evil sourcery.
@pks-t
azure: move build scripts into "azure-pipelines" directory
ffac520 
Since we have migrated to Azure Pipelines, we have deprecated and
subsequentally removed all infrastructure for AppVeyor and
Travis. Thus it doesn't make a lot of sense to have the split
between "ci/" and "azure-pipelines/" directories anymoer, as
"azure-pipelines/" is essentially our only CI.

Move all CI scripts into the "azure-pipelines/" directory to have
everything centrally located and to remove clutter in the
top-level directory.
@pks-t
azure: fix building in MinGW via Bash
d8e85d5 
Azure Pipelines supports bash tasks on Windows hosts due to it always
having Git for Windows included. To support this, the Git for Window
directory is added to the PATH environment to make the bash shell
available for execution. Unfortunately, this breaks CMake with the MinGW
generator, as it has sanity checks to verify that no bash executable is
in the PATH. So we can either remove Git for Windows from the path, but
then we're unable to execute bash jobs. Or we can add it to the path,
but then we're unable to execute CMake with the MinGW generator.

Let's re-model how we set the PATH environment. Instead of setting up
PATH for the complete build job, we now set a variable "BUILD_PATH" for
the job. This variable is only being used when executing CMake so that
it encounters a sanitizied PATH environment without GfW's bash shell.
@pks-t
azure: replace mingw setup with bash
443df2d 
We're about to phase out our Powershell scripts as Azure
Pipelines does in fact support Bash scripts on all platforms. As
a preparatory step, let's replace our MinGW setup script with a
Bash script.
@pks-t
azure: explicitly specify CMake generator
8e356f4 
We currently specify the CMake generator as part of the CMAKE_OPTIONS
variable. This is fine in the current setup, but during the conversion
to drop PowerShell scripts this will prove problematic for all
generators that have spaces in their names due to quoting issues.

Convert to use an explicit CMAKE_GENERATOR variable that makes it easier
to get quoting right.
@pks-t
azure: avoid executing compiler if there is none
1be4f89 
Until now, we always had the CC variable defined in the build.sh
pipeline. But as we're about to migrate the Windows jobs to Bash, as
well, those will not have CC defined and thus we cannot use "$CC" to
determine the compiler version.

Fix this by only executing "$CC" if the variable was set.
@pks-t
azure: use bash scripts across all platforms
0cda525 
Right now, we maintain semantically equivalent build scripts in
both Bash and Powershell to support both Windows and non-Windows
hosts. Azure Pipelines supports Bash on Windows, too, via Git for
Windows, and as such it's not really required to maintain the
Powershell scripts at all.

Remove them to reduce our own maintenance burden.
@pks-t
azure: skip SSH tests on Win32 platforms
f867bfa 
On Win32 build hosts, we do not have an SSH daemon readily available and
thus cannot perform the SSH tests. Let's skip the tests to not let Azure
Pipelines fail.
@pks-t
azure-pipelines: make gitdaemon tests work on Win32
415ee61 
On Win32 builds, the PID file created by git-daemon contained in invalid
PID that we were not able to kill afterwards. Somehow, it seems like the
contained PID was wrapped in braces. Consequentially, kill(1) failed and
thus caused the build to error.

Fix this by directly grabbing the PID of the spawned git-daemon process.
@pks-t pks-t force-pushed the pks/azure-drop-powershell branch from 9384a7c to 415ee61 Compare July 20, 2019 17:10
@pks-t pks-t merged commit 82b1d1d into libgit2:master Jul 21, 2019
@pks-t pks-t deleted the pks/azure-drop-powershell branch July 21, 2019 10:25
@pks-t
Copy link
Member Author

pks-t commented Jul 21, 2019

Thanks a lot for your review, @ethomson! :)

@implausible implausible mentioned this pull request Jul 23, 2019
Merged
@snyk-bot snyk-bot mentioned this pull request Feb 23, 2020
Open
@snyk-bot snyk-bot mentioned this pull request Apr 22, 2020
Open
@snyk-bot snyk-bot mentioned this pull request May 5, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ethomson ethomson ethomson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@pks-t @ethomson