Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu package #52

Open
dstillman opened this issue Mar 13, 2017 · 22 comments
Open

Ubuntu package #52

dstillman opened this issue Mar 13, 2017 · 22 comments
Labels

Comments

@dstillman
Copy link
Member

@dstillman dstillman commented Mar 13, 2017

With the switch to a single standalone version, it's possible we should put out an official Ubuntu package. I have no idea what's involved with this, but https://forums.zotero.org/discussion/25317/install-zotero-standalone-from-ubuntu-linux-mint-ppa/p1 and https://github.com/smathot/zotero_installer are possibly relevant.

I'm not going to set this up (other than creating an official Zotero account somewhere if necessary), but if it can be made simple enough I'm happy to add it to the existing deploy scripts.

@dstillman

This comment has been minimized.

Copy link
Member Author

@dstillman dstillman commented Apr 4, 2017

Or, alternatively, package the Linux version using Flatpak or AppImage.

@nedrichards

This comment has been minimized.

Copy link
Contributor

@nedrichards nedrichards commented Jan 31, 2018

there's currently a flatpak package of this on flathub - although a native one from source code would be better. As per flathub policy, if you'd like to be responsible, as the developer you'd be more than welcome.

@Eothred

This comment has been minimized.

Copy link

@Eothred Eothred commented Jun 5, 2018

I added a shell script that creates the appimage on a fork. It essentially depends on 'convert' (for the icon) and wget which are available on most systems I expect. It isn't very properly written (so I will not make a pull request of this in the current state). Someone could probably easily take those steps and write it in a way that it is more properly integrated into the zotero build system.

From the top of my head I know of canoe as another project using npm that builds appimages as releases. (Gruntfile? I know very little about this system in general, sorry) Maybe one can see how it is built there.

@howthebodyworks

This comment has been minimized.

Copy link

@howthebodyworks howthebodyworks commented Jul 21, 2019

@retorquere maintains a repo of zotero debs.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Jul 21, 2019

.deb packaging is easy if you have access to the files as laid out just before they'll be tarballed-up., and if you're OK with what is known as a "simple repo" my scripts are pretty stable now, but the way they package Zotero they're not likely to ever be mainlined into Debian (which is the usual path to get into derivative distros like Ubuntu) because I just put Zotero on disk as it is tarballed, and the zotero-standalone-build fetches a binary (firefox ESR) -- Debian wants everything to be either build from source, or built on an existing debian dependency.

PPA packaging is a mess and is hard to automate. The only benefit is that it's an easier route to get into Ubuntu, but they also want from-source builds, so see former point.

My scripts would be simple to adapt for packaging as part of the build process -- and they'd be stripped pretty far down as they'd not have to detect new versions being available, they wouldn't need to be for both Zotero and Juris-M, etc. I'd be happy to relinquish hosting the .debs.

The only real thing that would remain is where to host the binaries -- I host them in github releases, but hosting them on S3 would be simpler still. The most important driver for the decision on where to host them is whether you want download stats I think. GH stats are not super great, I don't know about S3. SourceForge is also super easy to set up, and has better download stats (and it's allowed to only use them as a bin hosting service), but SF isn't as popular with the OSS crowd as they once were.

Bintray seemed like the most logical choice but it was finicky to set up, and I hit upload limits during the first two weeks of testing, and I didn't want to be hobbled by the risk that I'd need to put out a release but couldn't.

@howthebodyworks

This comment has been minimized.

Copy link

@howthebodyworks howthebodyworks commented Jul 21, 2019

@retorquere aside: Your various contributions to Zotero, especially BBT have saved days of my life, thank you.

Also, thanks for the high-speed explanation of debs, Emiliano! I wonder if we should be avoiding some of the friction that you identified by packaging it the way Ubuntu seems to want pre-built apps distributed these days, i.e. Snap packages. This process I think is not dissimilar to the flatpak builds, which one can also run on Ubuntu.

Snapcraft explicitly supports pre-built-binaries and seems to include hosting, so that sidesteps the hosting and build questions. It might have other frictions, however. I'm not sure if people are have strong commitments to having both formats/pros and cons?

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Jul 21, 2019

I have no strong opinion on snaps -- I tend to opt for debs when available but that's probably just because I'm used to them. To be clear, debs support pre-build binaries, it's just that PPAs/Debian build rules don't allow them (which is why e.g. Oracle Java is packaged as a shell script that downloads and installs Java during .deb installation).

Snap tools are likely to be more pleasant to work with, and I say this not because I know them but because I can't imagine them being equally frustrating as the .deb tools. It took me ages to find out what was in the end very simple; there's a thousand ways to package debs, and a lot of the docs and tutorials tend to

  1. want to explain everything so even the most complex cases are covered ("If you wish to make an apple pie from scratch, you must first invent the universe"), and
  2. be outdated

It is not a pleasant process, but once done, it's set and forget. Because I still want to verify I'm releasing the right thing I'm releasing "manually" but that's really just a commit of an auto-updated config file (and that would be skipped for a real Zotero build, because the check I do is just whether it indeed grabbed the correct version); Travis takes care of the actual build & repo update.

One upside of debs vs snaps is that Chromebook users can install those. Other than that, anything that reliably gets me the latest version of Zotero is fine by me. I have no preference one way or the other.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Jul 21, 2019

I did have at one point a setup that used a fork of zotero-standalone-build to build a PPA-compatible package from scratch, but the PPA build infra blocks network access during build, so you can't fetch the firefox binary as part of the build.

Since you can't fetch binaries during build, and you're not allowed to have them as "source", PPAs/Debian mainline will just not work. The same goes for anything that relies on npm packages -- they must be existing Debian/Ubuntu packages. I don't see mainlining ever happening for this reason, because it will affect the electron build in the same way. (edit: although technically you could check in the node_modules directory to get around this, but that's yucky)

Self-hosted debs don't have this problem, and neither do snaps/flatpaks.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

I can convert (simplify, mainly) my existing script for https://github.com/retorquere/zotero-deb, but I'd need to know

  • what is the build environment for releases? Packaging debs requires a debian-based system to create the packages. I currently build on Travis.
  • what is the stage in the build where the client is laid out on disk as it is about to be tarballed and where is it laid out? Or should I just work from the tarball (not a problem, that's what I do now)
  • are you OK with a "simple repo" built with apt-ftparchive?
  • where do you intend to host the repo (S3, GH release, something else)?

(edit: downside of a simple repo is that there's no multiple versions for separate distro-versions, but Zotero doesn't have this in any case)

@dstillman

This comment has been minimized.

Copy link
Member Author

@dstillman dstillman commented Aug 6, 2019

what is the build environment for releases?

Not Debian-based, but we could use Docker for that part.

what is the stage in the build where the client is laid out on disk as it is about to be tarballed and where is it laid out?

Files are available in dist/ ($DEST_DIR) before the tarball step.

are you OK with a "simple repo" built with apt-ftparchive?

I'm not sure what that means.

where do you intend to host the repo (S3, GH release, something else)?

The repo in this case is…the thing that gets checked for updates? What form does that take? We tend to keep update manifests on actual webservers in case we need to do conditional updates for any reason (e.g., not serving an update to a particular set of clients). Downloadable files would go on S3 with the rest of the binaries. But I can handle adding the upload stuff as long as I know what files need to go where.

Thanks for working on this!

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

Files are available in dist/ ($DEST_DIR) before the tarball step.

But they wouldn't be in the Docker env, right? The scripts inside docker could just fetch the tarball as I do now.

are you OK with a "simple repo" built with apt-ftparchive?

I'm not sure what that means.

It means that given a Zotero arch/version, there's only one .deb, not one for each Debian/Ubuntu/whatnot release. In a full repo, you can have one version for Ubuntu 18.10, one for 18.04, etc, but I tried setting that up and it was a major PITA. Given that Zotero only has a single build per arch/version, I don't see any benefit in trying.

where do you intend to host the repo (S3, GH release, something else)?

The repo in this case is…the thing that gets checked for updates?

Correct, and where the debs are downloaded from

What form does that take?

Any web server that supports https and that allows files to be downloaded directly (although 301/302 redirects are OK) with a GET;

https://hostname/whatever/path/here/can/be/long

will work as long as

https://hostname/whatever/path/here/can/be/long/InRelease
https://hostname/whatever/path/here/can/be/long/Packages.bz2

etc will work; eg, a simple website hosted in an S3 bucket will work

You can see a full list of assets at https://github.com/retorquere/zotero-deb/releases/tag/apt-get for a simple repo; obviously, a Zotero repo would not host Juris-M binaries. I'd still be maintaining the repo for JM binaries until Frank decides he wants to host them himself, but I'd strip Zotero from the repo.

We tend to keep update manifests on actual webservers in case we need to do conditional updates for any reason (e.g., not serving an update to a particular set of clients).

A particular set of clients as in OS-dependent? Or another discriminator?

Downloadable files would go on S3 with the rest of the binaries.

A redirect to S3 would be OK (technically the assets on GH releases are hosted from S3 buckets, and you get sent there by a redirect), but simple repos (and as far as I can tell, full repos) require all URLs to sit on the same base; you can't have just https://hostname1/Packages and https://hostname2/zotero_5.0.73-1_amd64.deb

(the -1 at the end is a version bump so I can fix things in the packaging while keeping the binaries at the same version).

@dstillman

This comment has been minimized.

Copy link
Member Author

@dstillman dstillman commented Aug 6, 2019

But they wouldn't be in the Docker env, right?

No, but we could easily specify the dist directory as a mounted volume. While we could run this as a separate step after the normal tarball upload, there's no real reason to redownload and extract the same files.

Given that Zotero only has a single build per arch/version, I don't see any benefit in trying.

Right.

A particular set of clients as in OS-dependent? Or another discriminator?

OS version or current Zotero version, mainly. Though I doubt we'd care about distro/version here, and presumably we wouldn't get the Zotero version anyway. So this probably doesn't really matter.

(Basically, this has been useful in the past for things like not serving Zotero above a certain version to users on old versions of macOS, or serving a special manual-upgrade message to clients where the updater was broken.)

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

Does the build process build both 32 and 64 bit? To finalize the repo you need all packages in place.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

It's possible to run them one by one no issue, but

  1. You'd need to have a staging area where they come together, and
  2. You don't want the repo-refresh after the package build to be done in parallel.
@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

Can I detect in the dist directory what architecture (32/64 bit) was being built?

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

(other than by inspecting the ELF header of zotero-bin)

@dstillman

This comment has been minimized.

Copy link
Member Author

@dstillman dstillman commented Aug 6, 2019

After the done here, there'll be Zotero_linux-i686 and Zotero_linux-x86_64 directories.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

OS version or current Zotero version, mainly. Though I doubt we'd care about distro/version here, and presumably we wouldn't get the Zotero version anyway. So this probably doesn't really matter.

(Basically, this has been useful in the past for things like not serving Zotero above a certain version to users on old versions of macOS

I hate to say this, but this is possible using a full repo. You can have varying versions per target distro. It was a major PITA to get even halfway though, I didn't get it to work last time, and the docs on the process are not set up for easy digestion. Tutorials abound, many of them outdated and conflicting.

or serving a special manual-upgrade message to clients where the updater was broken.)

Broken updaters are out of scope for debs -- Zotero-updating is disabled in the debs because they're installed as root, updates would come from the dep repo.

After the done here, there'll be Zotero_linux-i686 and Zotero_linux-x86_64 directories.

Ah but that script has $arch so it could just pass that in.

@dstillman

This comment has been minimized.

Copy link
Member Author

@dstillman dstillman commented Aug 6, 2019

I hate to say this, but this is possible using a full repo.

That's OK. There's essentially no chance we'd worry about old distro compatibility the way we've had to worry about macOS and Windows compatibility.

Broken updaters are out of scope for debs

Right.

If we do decide to split things up and use redirects to S3 for the debs, I'll deal with it. For now we can assume they'll just go to the same place.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

I hate to say this, but this is possible using a full repo.

That's OK. There's essentially no chance we'd worry about old distro compatibility the way we've had to worry about macOS and Windows compatibility.

You have no idea what a relief this is 😆

If we do decide to split things up and use redirects to S3 for the debs, I'll deal with it. For now we can assume they'll just go to the same place.

That's cool. Then it's mostly done -- at the done point there's Zotero_linux-i686 and Zotero_linux-x86_64? Not "or"?

edit: I see now, it is indeed "and"

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

Alright, the current script can be started after done; when done, it will have apt/repo ready to upload. a GPG key needs to be present, I've generated mine using this.

@retorquere

This comment has been minimized.

Copy link

@retorquere retorquere commented Aug 6, 2019

"the current script" being #73

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.