Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux: add support for building snap packages #436

Merged
merged 6 commits into from Sep 11, 2019
Merged

Conversation

ppd
Copy link
Member

@ppd ppd commented May 31, 2019

This is a "copy & paste" integration of the solvespace snap repo @ https://github.com/ppd1990/solvespace-snap as discussed in ppd/solvespace-snap#2

A few things left to discuss

  1. The integration resides in $ROOT/snap. This is what build.snapcraft.io requires. If using the Ubuntu provided build server (and thus easy auto-build on commit) is not wanted, we could move it to mirror the flatpak integration

  2. The svg icon should probably find its place in res/freedesktop or we could just use a png icon. At the moment snap packages need an absolute path to an icon.
    The svg icon was proposed for inclusion in freedesktop: Add svg icon #441

  3. The snap is declared grade: stable. Maybe that should only apply to tagged versions. Some extra logic would be required

The next step

Publishing of the snap should be done under the flag of the upstream project. Therefore I propose a name transfer to a newly created "SolveSpace" publisher. This has to be requested on forum.snapcraft.io.

Testing the build process

# optional for a faster build (adjust values for virtual build machine)
export SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=4G
export SNAPCRAFT_BUILD_ENVIRONMENT_CPU=6

snapcraft
...
snap install --dangerous solvespace_3.0~9ac55f39_amd64.snap 

@CLAassistant
Copy link

CLAassistant commented May 31, 2019

CLA assistant check
All committers have signed the CLA.

@ppd ppd mentioned this pull request May 31, 2019
@whitequark
Copy link
Contributor

whitequark commented Jun 11, 2019

  • The integration resides in $ROOT/snap. This is what build.snapcraft.io requires. If using the Ubuntu provided build server (and thus easy auto-build on commit) is not wanted, we could move it to mirror the flatpak integration

Hmm that's a bit annoying. Maybe both flatpak and snap packaging should be moved out into separate repositories? But then auto-build on commit still wouldn't work when a new commit to solvespace/solvespace is made. I'm not sure how to address this.

  • The svg icon should probably find its place in res/freedesktop or we could just use a png icon. At the moment snap packages need an absolute path to an icon.

An svg icon for freedesktop platforms would be very nice. Would you be able to do this as a separate PR? There would be no problems merging that.

3. The snap is declared grade: stable. Maybe that should only apply to tagged versions. Some extra logic would be required

Yes ideally that would be done. The Travis CI configuration already has the necessary logic, but that presumes we'll build with Travis, which is not currently actually possible, since Travis builds turn GUI off due to an extremely outdated GTK there.

@ppd
Copy link
Member Author

ppd commented Jun 11, 2019

Hmm that's a bit annoying. Maybe both flatpak and snap packaging should be moved out into separate repositories? But then auto-build on commit still wouldn't work when a new commit to solvespace/solvespace is made. I'm not sure how to address this.

Documentation exists to build snap packages from a Docker container under Travis(https://tutorials.ubuntu.com/tutorial/continuous-snap-delivery-from-travis-ci). So skipping the build server is probably an option.

Another option: Keep the snap integration in a separate repo and just trigger an update into a file in that repo like date --iso-8601=seconds >> TRIGGERED_UPDATES && git [...]. That would be quite loosely coupled.

An svg icon for freedesktop platforms would be very nice. Would you be able to do this as a separate PR? There would be no problems merging that.

Can do.

@ppd ppd mentioned this pull request Jun 14, 2019
@whitequark
Copy link
Contributor

whitequark commented Jun 14, 2019

Documentation exists to build snap packages from a Docker container under Travis (https://tutorials.ubuntu.com/tutorial/continuous-snap-delivery-from-travis-ci). So skipping the build server is probably an option.

I'd say: let's put it under pkg/snap and use Docker to build on Travis CI.

@ppd
Copy link
Member Author

ppd commented Jun 14, 2019

pkg/snap seems a bit problematic due to this reason:

snapcraft mounts the working directory into the build container like so:

$ mount
[...]
/home/mfederle/Desktop/solvespace/pkg on /root/project type fuse.sshfs [...]

Thus, we cannot access the solvespace src with a relative path in snapcraft.yaml.

Options?

  1. Don't move into pkg
  2. Use a git URL in snapcraft.yaml. Not very elegant
  3. Separate repository

@whitequark
Copy link
Contributor

whitequark commented Jun 15, 2019

That's uh, pretty obnoxious on the part of snapcraft. I guess I'll have to go through every option it has with a fine-toothed comb...

@ppd
Copy link
Member Author

ppd commented Jun 15, 2019

It's very clearly intended to be in a separate repo or to reside in the first-level subdirectory snap. I guess that's also KISS in a way...

Here is an example of a snap that uses a build-trigger file to fire the build:
https://github.com/snapcrafters/android-studio

@whitequark
Copy link
Contributor

whitequark commented Jun 15, 2019

Well, neither of Canonical's supposed solutions works for me, and this isn't an issue with pretty much any other packaging system, so...

@whitequark
Copy link
Contributor

whitequark commented Jun 15, 2019

If there is absolutely no way to get snapcraft to refer to .., then I suggest a wrapper script that uses git worktree add pkg/snap/worktree to make an additional checkout, and removes it once everything's done. Would that work?

@ppd
Copy link
Member Author

ppd commented Jun 15, 2019

A worktree does not work with the snapcraft.yaml as it is, because the worktree references a path outside of the virtual machine:

snapcraft-solvespace ../parts/solvespace/src# cat .git
gitdir: /home/mfederle/Desktop/solvespace/.git/worktrees/solvespace-worktree

This then fails the git commands.

What works, is to just clone from the parent path like so:

git clone .. solvespace-snap-src

That's certainly an option for staying close to the paradigm of not generating snapcraft.yaml with cmake, but containing the snapcraft specific logic as much as possible within snapcraft.yaml.

I'll investigate whether it is maybe even possible to somehow contain the setup and cleanup commands in snapcraft.yaml itself.

@whitequark
Copy link
Contributor

whitequark commented Jun 15, 2019

Thank you!

@ppd
Copy link
Member Author

ppd commented Jun 15, 2019

I think the build script is a sensible choice.

As I understand it, the build environment for snapcraft is basically either host or multipass, where host would apply to the Docker case, and thus to Travis. So maybe the relative path would even work in such an environment, but the resulting snapcraft.yaml wouldn't be environment-agnostic and hardly well-formed.

@ppd
Copy link
Member Author

ppd commented Jun 15, 2019

The fun continues: https://hub.docker.com/r/snapcore/snapcraft is the official snapcraft docker image and it is still based on Ubuntu 16.04 which makes it a bit useless for compiling base: core18 snaps.

I'll investigate that further.

I adapted the official snapcraft Dockerfile to work with base: core18 and tweaked it a little to preinstall most build-packages: https://github.com/ppd1990/solvespace-snapcraft-docker

The result is published as mfederle/solvespace-snapcraft:latest.

Building with docker works like this:

cd solvespace
sudo docker run -v $(pwd):/solvespace mfederle/solvespace-snapcraft:latest sh -c "apt update && sh /solvespace/pkg/snap/build.sh"

Beware: Building like this will pollute pkg/snap with build artifacts.

Bringing Travis into the picture

snapcraft enable-ci travis can be used to set up snapcraft in .travis.yml. One minor adjustment needed, is of course the above mentioned Docker image instead of the official one.

Maybe that functionality belongs in another pull request entirely. What do you think?

@ppd ppd force-pushed the snap branch 2 times, most recently from d15b571 to a630824 Compare Jun 22, 2019
@ppd
Copy link
Member Author

ppd commented Jun 25, 2019

Another option: Run a Travis job that syncs this repo's snapcraft.yaml to another repository whose sole function is to serve as a source for build.snapcraft.io.

In the snapcraft.yaml itself we'd add support for building from the source tree itself (i.e. local) as well as from the git repository.

@whitequark
Copy link
Contributor

whitequark commented Jul 2, 2019

What do you think?

Sorry, I missed your edit. I'll take a look soon!

@ppd
Copy link
Member Author

ppd commented Jul 10, 2019

Note to ourselves:

Snapcraft supports remote builds on Launchpad: https://forum.snapcraft.io/t/preview-snapcraft-remote-build/11541

A possible pipeline: Github -> Travis --- (Snapcraft) ---> Launchpad => snaps for multiple architectures

This would offload all building again and of course the issue of compiling for other architectures (if so desired) is not our concern.

@whitequark
Copy link
Contributor

whitequark commented Jul 10, 2019

I think that's a reasonable option. What should we do for that? Separate repo, an encrypted SSH key on Travis, and trigger builds from there?

@ppd
Copy link
Member Author

ppd commented Jul 23, 2019

Oh. I'm sorry I haven't replied yet. I'll investigate and create a prototype as soon as I'm not swamped with work any more...

@ppd
Copy link
Member Author

ppd commented Aug 25, 2019

Coming back to this, a few observations:

  1. Custom docker images won't be necessary for snapcraft work as parametrized docker images are being proposed atm: docker: parametrize risk and OS for image builds (core18 etc) snapcore/snapcraft#2673

  2. Remote build is still in beta: https://forum.snapcraft.io/t/preview-snapcraft-remote-build/11541/10?u=ppd

I think that's a reasonable option. What should we do for that? Separate repo, an encrypted SSH key on Travis, and trigger builds from there?

The SSH key and the snapcraft credentials must be stored on Travis.

From the snapcraft example:

openssl aes-256-cbc -K <travis-key> -iv <travis-iv>
      -in .snapcraft/travis_snapcraft.cfg
      -out .snapcraft/snapcraft.cfg -d

Build & release then something like this:

docker run -v $(pwd):/solvespace mfederle/solvespace-snapcraft:latest sh -c 
"apt update && sh /solvespace/pkg/snap/build.sh" &&
snapcraft push *.snap --release edge

To summarize, we need:

  1. A snapcraft version with remote-build enabled
  2. Launchpad credentials in Travis
  3. Snapcraft cfg in Travis

I'll report back when I have a usable Travis cfg.

@ppd
Copy link
Member Author

ppd commented Aug 25, 2019

@whitequark I have a sample travis config at https://github.com/ppd1990/solvespace-travis that makes use of snapcraft remote-build and this branch

Builds are successful. Only the deploy step in Travis threw me an error, but that's probably a temporary issue on the server side... Deploy also works.

Build log: https://travis-ci.org/ppd1990/solvespace-travis/builds/576488983

@ppd
Copy link
Member Author

ppd commented Aug 28, 2019

Note regarding maintainability: One of the very next snapcraft releases will ship the gnome/desktop extension and will thus render lots of the fluff (the extra part, those gtk-common-themes plugs...) in snapcraft.yaml obsolete, making everything look a lot cleaner.

@whitequark
Copy link
Contributor

whitequark commented Aug 28, 2019

Is that happening soon? If so we might just wait a bit.

@ppd
Copy link
Member Author

ppd commented Aug 28, 2019

I guess so. Seems like they do about one release per month and the last non-minor release was on 2019-07-23.

We're also still waiting for the remote-build functionality that is still not in a stable release of snapcraft. So we might soon get the new release but the remote-build branch of snapcraft is not guaranteed to instantly rebase on this release.

@ppd
Copy link
Member Author

ppd commented Sep 10, 2019

Implemented changes:

  • added SNAP CMake option to install snap-specific desktop file
  • Install icons in correct location in preparation for when snapd gains icon theme support
  • Replace git clone with rsync in build.sh to allow building uncommited code

@whitequark
Copy link
Contributor

whitequark commented Sep 10, 2019

Looks great. The only blocker I see right now is grade; we can merge this PR and add remote builds when they're available.

@whitequark
Copy link
Contributor

whitequark commented Sep 10, 2019

Can you adjust rsync so that it only syncs the files in git, or at least skips the build folder? Mine is 5 GB in size.

@ppd
Copy link
Member Author

ppd commented Sep 10, 2019

Can you adjust rsync so that it only syncs the files in git, or at least skips the build folder? Mine is 5 GB in size.

Oh yeah. That is a rather naive implementation so far. I'll look into it.

Looks great. The only blocker I see right now is grade

What's your preferred strategy? grade = devel mainly prevents you from releasing to a stable channel. That's about it.

@whitequark
Copy link
Contributor

whitequark commented Sep 10, 2019

Oh yeah. That is a rather naive implementation so far. I'll look into it.

Looks like --filter=':- .gitignore' works

@whitequark
Copy link
Contributor

whitequark commented Sep 10, 2019

What's your preferred strategy? grade = devel mainly prevents you from releasing to a stable channel. That's about it.

Ideally check if the commit corresponds to a tag, and select grade based on that.

@whitequark
Copy link
Contributor

whitequark commented Sep 10, 2019

Is it possible to make the CLI install as solvespace.cli instead of solvespace.solvespace-cli?

@ppd
Copy link
Member Author

ppd commented Sep 10, 2019

That's easy. I'll update the PR soon

@ppd
Copy link
Member Author

ppd commented Sep 11, 2019

I addressed the requests and also added license info to the snapcraft declaration.

One thing that still bugs me a bit: The hardcoded version scheme 3.0~$hash. We could extract it from CMakeLists.txt instead, but that is a bit hacky. I'd love to use git describe or similar, but given that we have no tag like 3.0-dev or similar, that is not really an option.

@ppd ppd changed the title WIP: Linux: add support for building snap packages Linux: add support for building snap packages Sep 11, 2019
pkg/snap/snap/snapcraft.yaml Outdated Show resolved Hide resolved
@whitequark whitequark merged commit 219df31 into solvespace:master Sep 11, 2019
@whitequark
Copy link
Contributor

whitequark commented Sep 11, 2019

Thanks.

@ppd ppd deleted the snap branch Sep 11, 2019
@ppd ppd mentioned this pull request Sep 11, 2019
@whitequark
Copy link
Contributor

whitequark commented Nov 26, 2019

I'm curious: why is the network plug enabled? SolveSpace shouldn't really be talking over network.

@ppd
Copy link
Member Author

ppd commented Nov 26, 2019

The reason is a bit convoluted: By default, snaps will not have access to exotic PAM providers like SSSD and thus will not recognize "non-traditional" users like those configured via LDAP etc.

A workaround, as proposed in https://forum.snapcraft.io/t/accessing-unix-users-and-groups-from-a-strict-snap/9035/7, is to install a user info proxy on the machine like nscd.
To access that proxy's socket within the confinement, we need the network plug.

@whitequark
Copy link
Contributor

whitequark commented Nov 26, 2019

OK but ... SolveSpace doesn't care one bit about UNIX groups and users.

@whitequark
Copy link
Contributor

whitequark commented Nov 26, 2019

Oh right, we do care about $HOME and that's from passwd and we don't use the Portal API because we have to access a file at a relative path for assemblies. Sigh.

@ppd
Copy link
Member Author

ppd commented Nov 26, 2019

Solvespace doesn't, but its desktop integration does. The gnome-3-28 extension bit sets up some softlinks that enable the user to see their "real" home folder and such from the file management dialogs despite it having its own $HOME within confinement.

+ What you wrote

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants