Set up automated builds for LDC snap #21

WebDrake · 2017-03-16T19:11:41Z

Ubuntu is now offering an automated build service for snaps, which will automatically build and push out a new revision of the snap package whenever the master branch is updated:
https://build.snapcraft.io/

This will require authorizing some GitHub hooks via my account and on behalf of the ldc-developers group. @klickverbot @JohanEngelen is it OK to move forwards with this?

WebDrake · 2017-03-16T19:54:35Z

As a secondary question: what platform architectures are currently supported by LDC (in terms of actually running on those platforms)? I imagine amd64 and x86, but any others ... ?

WebDrake · 2017-03-17T22:47:22Z

This will require authorizing some GitHub hooks via my account and on behalf of the ldc-developers group. @klickverbot @JohanEngelen is it OK to move forwards with this?

Ping on this? I'm asking simply because I don't feel comfortable asking a 3rd-party webapp to hook into the ldc-developers group without an OK from others.

dnadlinger · 2017-03-17T23:04:46Z

Sure, let's go ahead with this – just didn't notice this earlier.

WebDrake · 2017-03-17T23:29:46Z

Hmmm, well, this ran into a brick wall quickly :-P They only detect/handle repos under my own handle, despite giving them access to the ldc-developers group. I'll follow up and see what can be done ...

WebDrake · 2017-03-17T23:51:54Z

In the short term, one option would be to use Launchpad to build and push the packages. Would it be OK to set up an ldc-developers team there? (I'd rather not just set up a project via my own launchpad account, although that's easy to do if need be.)

dnadlinger · 2017-03-18T00:25:38Z

I can't see any potential issues with that either.

WebDrake · 2017-03-18T11:06:29Z

OK. There's now a new ldc-developers group on Launchpad: https://launchpad.net/~ldc-developers

... which owns a new ldc2 snap package project: https://launchpad.net/ldc2.snap

... which has a git clone of the present GitHub repo: https://code.launchpad.net/~ldc-developers/ldc2.snap/+git/ldc2.snap

... which is set up to automatically build a new snap package every time the master branch is updated (it pulls every 5 hours from the github repo): https://code.launchpad.net/~ldc-developers/+snap/ldc2

Currently it looks like i386 builds are failing, but I think that's a known issue that should be fixed shortly (https://bugs.launchpad.net/snapcraft/+bug/1665165). The amd64 build is proceeding fine so far.

Are there any other processors it would be worth building for? Those available (besides amd64 and i386) are arm64, armhf, powerpc, powerpc64el and s390x.

WebDrake · 2017-03-18T11:19:09Z

There's now a new ldc-developers group on Launchpad

BTW if anyone else from the LDC team wants to sign up on Launchpad (or is already), I can add them to the members of this group, and invite them as collaborators on the snap package. Just in case an elephant falls on me at DConf, or something like that.

This is necessary in order to ensure that CI tools can access git source repos. In support of ldc-developers#21.

dnadlinger · 2017-03-18T12:47:32Z

My account name there is the same as here. (For whatever reason, I seem to have ended up with two accounts after the whole Ubuntu One login thing.)

WebDrake · 2017-03-18T12:58:29Z

OK, you should be in the team now. An amd64 build of the updated package is now in progress now that the git/https URL issue is fixed.

This should probably save some time as well as avoiding the need for the user to have git installed. It should also work around a problem with the Launchpad build setup not being able to clone a git repo. In support of ldc-developers#21.

This will hopefully resolve the Launchpad built system's issues when trying to clone the git repositories. In support of ldc-developers#21.

WebDrake · 2017-03-18T21:22:37Z

I've run into a couple of slightly unpleasant issues with the Launchpad build service which are getting in the way of building the package successfully.

the build service only supports https:// URLs when cloning git repos (see: https://bugs.launchpad.net/launchpad-buildd/+bug/1663920), and LDC's .gitmodules file uses git:// instead. I can send a patch to address this, but presumably that would require a further patch release for me to be able to use it.
I could use the source tarballs that can be downloaded directly from GitHub (cf. Replace git downloads with downloads of source tarball #23), but this runs into a rather subtle issue: because the snap package itself is git-hosted, the CMakeLists rules to check the git describe output clash with the stated LDC_VERSION, and I wind up with the hash ID of the latest snap package commit (!) being included in the version number of the bootstrap compiler. Are there any ready ways to prevent the git describe output being taken into account?

Any thoughts on what would be the best way forward (with least inconvenience to LDC developers)?

WebDrake · 2017-03-18T21:24:45Z

N.B. there is a 3rd option, which is to wait for the Launchpad devs to fix https://bugs.launchpad.net/launchpad-buildd/+bug/1663920.

This should probably save some time as well as avoiding the need for the user to have git installed. It should also work around a problem with the Launchpad build setup not being able to clone LDC's git submodules (since they use `git://` URLs instead of `https://`). In support of ldc-developers#21.

This option allows users to override the default behaviour of comparing `LDC_VERSION` to the output of `git describe` and preferring the latter in the case of any difference. This should obviously be used with caution but allows the user to avoid issues that can arise when e.g. building from a source tarball that has been extracted in a subdirectory of an independent git tree (!). This is unfortunately a real possibility: apart from folks who use git to version their home directory [1], it's also proven an issue when trying to use source tarballs to build the LDC snap package [2]. [1]https://stackoverflow.com/questions/21266948/use-git-to-manage-home-directory/42946460 [2]ldc-developers/ldc2.snap#21 (comment) This provides a workaround for ldc-developers#2042.

Switching from `git://` URLs to `https://` should make it easier to avoid some CI-related issues encountered with the LDC snap package: ldc-developers/ldc2.snap#21 (comment)

Since this function is only ever invoked from the base directory of the LDC source tree, it does not need to recurse in order to get the correct git revision for LDC. This patch therefore adds a parameter that can be set to `TRUE` or `FALSE` depending on whether one wants to use recursion or not, and sets it to false in when calling `get_git_head_revision` in LDC's own `CMakeLists.txt`. This should prevent issues that arise when building from source tarballs that have been extracted into a subdirectory of another git tree (!). This is unfortunately a real possibility: apart from folks using git to version their home directory [1], it has also proven an issue when using source tarballs to build the LDC snap package [2]. [1]https://stackoverflow.com/questions/21266948/use-git-to-manage-home-directory/42946460 [2]ldc-developers/ldc2.snap#21 (comment) Fixes ldc-developers#2042.

Since this function is only ever invoked from the base directory of the LDC source tree, it does not need to recurse in order to get the correct git revision for LDC. This patch therefore adds a parameter that can be set to `TRUE` or `FALSE` depending on whether one wants to use recursion or not, and sets it to false in when calling `get_git_head_revision` in LDC's own `CMakeLists.txt`. This should prevent issues that arise when building from source tarballs that have been extracted into a subdirectory of another git tree (!). This is unfortunately a real possibility: apart from folks using git to version their home directory [1], it has also proven an issue when using source tarballs to build the LDC snap package [2]. [1]https://stackoverflow.com/questions/21266948/use-git-to-manage-home-directory/42946460 [2]ldc-developers/ldc2.snap#21 (comment) Fixes #2042.

WebDrake · 2017-03-30T22:03:37Z

Latest ltsmaster and release-1.2.x branches now work with the Launchpad build system; I have successfully built amd64 and i386 patches from the devel branch. I will start publishing these builds via the edge channel once 0.17.4 and 1.2.0-beta2 are tagged.

Since this function is only ever invoked from the base directory of the LDC source tree, it does not need to recurse in order to get the correct git revision for LDC. This patch therefore adds a parameter that can be set to `TRUE` or `FALSE` depending on whether one wants to use recursion or not, and sets it to false in when calling `get_git_head_revision` in LDC's own `CMakeLists.txt`. This should prevent issues that arise when building from source tarballs that have been extracted into a subdirectory of another git tree (!). This is unfortunately a real possibility: apart from folks using git to version their home directory [1], it has also proven an issue when using source tarballs to build the LDC snap package [2]. [1]https://stackoverflow.com/questions/21266948/use-git-to-manage-home-directory/42946460 [2]ldc-developers/ldc2.snap#21 (comment) Fixes #2042.

WebDrake · 2017-03-30T23:20:50Z

OK, so, both ARM builds failed (not entirely a surprise), but for different reasons.

armhf failed because the custom C flags used are not supported:

[ 69%] Building C object runtime/CMakeFiles/druntime-ldc.dir/druntime/src/core/stdc/errno.c.o
cd /build/ldc2/parts/ldc/build/runtime && /usr/bin/cc  -DLDC_WITH_PGO -I/build/ldc2/parts/ldc/src/. -I/build/ldc2/parts/ldc/src/ddmd -I/build/ldc2/parts/ldc/src/ddmd/root -I/build/ldc2/parts/ldc/build/ddmd -I/build/ldc2/parts/ldc/src -isystem /build/ldc2/parts/llvm/install/include  -O3 -DNDEBUG -Wa,-mrelax-relocations=no -fPIC   -o CMakeFiles/druntime-ldc.dir/druntime/src/core/stdc/errno.c.o   -c /build/ldc2/parts/ldc/src/runtime/druntime/src/core/stdc/errno.c
Assembler messages:
Error: unrecognized option -mrelax-relocations=no
runtime/CMakeFiles/druntime-ldc.dir/build.make:2765: recipe for target 'runtime/CMakeFiles/druntime-ldc.dir/druntime/src/core/stdc/errno.c.o' failed
make[2]: *** [runtime/CMakeFiles/druntime-ldc.dir/druntime/src/core/stdc/errno.c.o] Error 1
make[2]: Leaving directory '/build/ldc2/parts/ldc/build'

... while arm64 failed because of a static assert in std.math:

[ 59%] Building C object runtime/CMakeFiles/druntime-ldc-debug.dir/druntime/src/ldc/arm_unwind.c.o
/build/ldc2/parts/ldc-bootstrap/src/runtime/phobos/std/math.d(1705): Error: static assert  "Not implemented for this architecture"
runtime/CMakeFiles/phobos2-ldc-debug.dir/build.make:249: recipe for target 'runtime/std/math-debug.o' failed
make[2]: *** [runtime/std/math-debug.o] Error 1
CMakeFiles/Makefile2:429: recipe for target 'runtime/CMakeFiles/phobos2-ldc-debug.dir/all' failed
make[1]: *** [runtime/CMakeFiles/phobos2-ldc-debug.dir/all] Error 2

I'll have to look into whether it's possible to set different C flags for different architectures, but in any case, I think support for ARM is a slight long shot right now. Curious about what the std.math static assert is, though (one to check out in the next days...).

WebDrake · 2017-04-09T15:09:19Z

I've now had a bunch of successful automated builds and uploads via Launchpad, for both i386 and amd64, so I think this issue can reasonably be considered closed.

Where other architectures are concerned I think this can be addressed via separate PRs and issues. ARM builds probably need specific input/advice anyway.

WebDrake added enhancement question labels Mar 16, 2017

WebDrake added a commit to WebDrake/ldc2.snap that referenced this issue Mar 18, 2017

Use https URLs for git repository locations

da18a99

This is necessary in order to ensure that CI tools can access git source repos. In support of ldc-developers#21.

WebDrake mentioned this issue Mar 18, 2017

Use https URLs for git repository locations #22

Merged

WebDrake mentioned this issue Mar 18, 2017

Replace git downloads with downloads of source tarball #23

Closed

WebDrake mentioned this issue Mar 18, 2017

Explicitly specify git as source-type for ldc and ldc-bootstrap parts #24

Merged

This was referenced Mar 19, 2017

get_git_head_revision cmake function searches beyond base directory of source tree ldc-developers/ldc#2042

Closed

Use https URLs for git submodules ldc-developers/ldc#2051

Merged

WebDrake mentioned this issue Mar 23, 2017

Add option to avoid recursing in get_git_head_revision cmake function ldc-developers/ldc#2052

Merged

WebDrake closed this as completed Apr 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up automated builds for LDC snap #21

Set up automated builds for LDC snap #21

WebDrake commented Mar 16, 2017

WebDrake commented Mar 16, 2017

WebDrake commented Mar 17, 2017

dnadlinger commented Mar 17, 2017

WebDrake commented Mar 17, 2017

WebDrake commented Mar 17, 2017

dnadlinger commented Mar 18, 2017

WebDrake commented Mar 18, 2017

WebDrake commented Mar 18, 2017

dnadlinger commented Mar 18, 2017

WebDrake commented Mar 18, 2017

WebDrake commented Mar 18, 2017 •

edited

Loading

WebDrake commented Mar 18, 2017

WebDrake commented Mar 30, 2017

WebDrake commented Mar 30, 2017

WebDrake commented Apr 9, 2017

Set up automated builds for LDC snap #21

Set up automated builds for LDC snap #21

Comments

WebDrake commented Mar 16, 2017

WebDrake commented Mar 16, 2017

WebDrake commented Mar 17, 2017

dnadlinger commented Mar 17, 2017

WebDrake commented Mar 17, 2017

WebDrake commented Mar 17, 2017

dnadlinger commented Mar 18, 2017

WebDrake commented Mar 18, 2017

WebDrake commented Mar 18, 2017

dnadlinger commented Mar 18, 2017

WebDrake commented Mar 18, 2017

WebDrake commented Mar 18, 2017 • edited Loading

WebDrake commented Mar 18, 2017

WebDrake commented Mar 30, 2017

WebDrake commented Mar 30, 2017

WebDrake commented Apr 9, 2017

WebDrake commented Mar 18, 2017 •

edited

Loading