Add packaging for Fedora

[pauldix]

It would be good to have InfluxDB installable from the official Fedora package system. The OpenStack community is asking for this and they've told me that this is the first step to then getting it into the packages for all the other RedHat flavors.

[btashton]

I might be able to help with this since I am in the Fedora packing group (years ago I use to do more in Fedora). One of the biggest hurdles here is going to be how libraries are bundled in with the build. These will need to be packaged separately, ideally they would be packaged for epel as well.

[toddboom]

@btashton That would be a huge help - thanks! With the v0.9.0 release, we're going to be cutting the dependencies down drastically, so bundling might be a nonissue at that point. Do you mind if we reach out in the next few weeks once the release is closer?

[btashton]

For sure, I will take a look at what you have now this weekend and try to get a copr repo up https://copr.fedoraproject.org/

[btashton]

Are there any bundled dependencies I should be aware of besides boltdb? I was going to work on getting that packaged up first.

[otoolep]

BoltDB is not a dependency. Since InfluxDB is a Go project, a statically linked binary (except for libc, which should be on every machine) is all that is generated. BoltDB code part of that single binary.

[otoolep]

Check the packaging script for more details.

https://github.com/influxdb/influxdb/blob/master/package.sh

[btashton]

Sure, but as for formal packaging is a "library" i.e. when a security or other bug is found in "github.com/boltdb/bolt" A rebuild must be triggered on the influxdb package to resolve this. So while the boltdb package on Fedora is not used to build influxdb it is in effect a rebuild dependency.

[otoolep]

OK, that makes sense. In that case the entire Go standard library is a dependency. :-)

[corylanou]

This is a build dependency, not a deployment or runtime dependancy. While we try to use as much of the standard library as possible, there are several third party packages that are used in our code. Any one of them, if they had a security issue or bug, would trigger a rebuild.

You can get an idea of what external packages we use by running go list -json in each package directory. I'll work on putting together a script at some point to actually dig out and list them easily, so if anyone ever wants to know they can quickly reference it.

[btashton]

Sure, and golang is a build requirement. I'm not making this requirement up, just trying to help get the packaging process started :)
http://fedoraproject.org/wiki/PackagingDrafts/Go#Dependencies

Quick look shows these as likely "BuildRequires":

• github.com/boltdb/bolt
• code.google.com/p/go-uuid/uuid
• github.com/bmizerany/pat
• github.com/peterh/liner
• github.com/kimor79/gollectd
• github.com/BurntSushi/toml
• github.com/rakyll/statik/fs
• golang.org/x/crypto/bcrypt

Looks like I forgot to click submit on this, and it was just sitting in my browser

[corylanou]

@btashton No one was accusing you of making it up. We love that you are contributing and hope we are being helpful in return.

[btashton]

I did not take it that way at all, I also know the pain of dealing with distro packaging rules. That go command you listed is quite useful thanks.

[corylanou]

A friend of mine wrote this but it isn't quite working yet, but it's closer (still includes too much standard library).

go list -f '{{join .Deps "\n"}}' ./... | grep -vxFf <(go list std) | sort -u

As soon as we get that sorted out it will be really easy to show the external packages used. This is good knowledge for the development team as well to always have.

[corylanou]

Ok, got the script working:

go list std | sort -u > /tmp/std.lst
go list -f '{{join .Deps "\n"}}' ./...  | sort -u > /tmp/dep.lst
sort /tmp/std.lst /tmp/std.lst /tmp/dep.lst | grep -v influxdb | uniq -u

Yields this result:

code.google.com/p/go-uuid/uuid
code.google.com/p/log4go
github.com/BurntSushi/toml
github.com/bmizerany/pat
github.com/boltdb/bolt
github.com/kimor79/gollectd
github.com/peterh/liner
github.com/rakyll/statik/fs
golang.org/x/crypto/bcrypt
golang.org/x/crypto/blowfish

[btashton]

Ok just to track this:

Package	Status	Fedora Package Name	Notes
code.google.com/p/go-uuid/uuid	[In Repo ]	golang-googlecode-uuid-devel	Packing issue this is fixed by bug https://bugzilla.redhat.com/show_bug.cgi?id=1192652
code.google.com/p/log4go	[In Repo]	golang-googlecode-log4go-devel
github.com/BurntSushi/toml	[In Repo]	golang-github-BurntSushi-toml-devel
github.com/bmizerany/pat	[In Repo]	golang-github-bmizerany-pat-devel
github.com/boltdb/bolt	[In Copr Repo]
github.com/kimor79/gollectd	[In Repo]	golang-github-kimor79-gollectd-devel
github.com/peterh/liner	[In Copr Repo]
github.com/rakyll/statik/fs	[In Repo]	golang-github-rakyll-statik-devel
golang.org/x/crypto/bcrypt	[In Repo]	golang-golangorg-crypto-devel
golang.org/x/crypto/blowfish	[In Repo]	golang-golangorg-crypto-devel

[btashton]

I just packaged up boltdb and liner and created a copr repo for these efforts until the packages get approved. I am going to start on the influx package now.
https://copr.fedoraproject.org/coprs/bashton/influxdb/

[btashton]

Got a packaging issue fixed upstream for influxdb, with the packages I have in copr, everything should be in-place to start packaging InfluxDB. Now that you have an RC out I'll try to find some time this weekend to push the boltdb and liner packages for review, and then get a functioning spec done for InfluxDB.

[azrdev]

@btashton are you still up to this?

[btashton]

I am still up for this, but I was waiting for things to settle down a bit. I was thinking about getting the next release point packaged and then get the reset of the package in my copr repo. I still need to write a systemd script for it.

[azrdev]

the rpm package from https://influxdb.com/download/index.html#redhat-centos already contains systemd and sysV init scripts in /opt/influxdb/versions/0.9.4.1/scripts/

The postinstall script in that rpm does (among others)

# Systemd
if which systemctl > /dev/null 2>&1 ; then
    cp /opt/influxdb/versions/0.9.4.1/scripts/influxdb.service         /lib/systemd/system/influxdb.service
    systemctl enable influxdb

# Sysv
else
...

[rossmcdonald]

Just to make sure everyone is aware, we now have our own package repo hosted at 'https://repos.influxdata.com' which can be used with Fedora (right now using the RHEL configuration). I'm all for having InfluxDB added to the official Fedora repos as well, though.

[btashton]

@rossmcdonald So that has bundled packages which is not ideal for upstream. Part of my effort was packaging up the bits that were not already in Fedora. I will add it to EPEL as well once it is ready. Part of why I stopped was there was no clear upgrade path in the past, and I dont want to put out packages where the upgrade breaks peoples data. Going forward will there be a clear data migration path (even with yet another backend)?

[rossmcdonald]

@btashton Yes, the new storage engine should be the last big change prior to v1. Right now there is not a clear migration from <=0.9.4.2 to tsm (the new storage engine), but we are hoping to come out with a data migration tool that will enable conversion of the shards (b1 -> tsm, for example) with the 0.9.6 release.

My recommendation would be to wait until 0.9.5 (should be within the next few weeks depending on how testing goes), due to the flux with the storage engine. We are also updating the filesystem hierarchy with the next release, which could complicate things even further. Everything after 0.9.5 should be pretty close to what v1 will look like, though.

[beckettsean]

@btashton, to add to what @rossmcdonald said, shards using the b1/bz1 engine can co-exist with shards using the tsm1 engine, so the upgrade is still a drop-in replacement from 0.9.4 to 0.9.5. There will be some features (like incremental hot backups) that don't work until all shards are tsm1, but the conversion tool is already under way.

[btashton]

Alright, I will start to pick this up again then. With the new more lax bundle rules in Fedora, I will start packaging things up again and we can make a decision about when it is appropriate for it to drop into a release.

[ingvagabund]

Hi, although Fedora have relaxed dependency bundling I would still like to build golang projects from debundled dependencies. Does it make sense for influxdb to generate Godeps.json file or its equivalent (e.g. GLOCKFILE)? So I can be aware which commit of each dependency is used for building?

[toddboom]

@rossmcdonald would this be hard?

[rossmcdonald]

@toddboom I don't have a lot of experience in this area, so I can't say for sure. I think it would be much easier just to add Fedora support to the repos.influxdata.com repository instead (which it kind of has already, I just need to tweak a few things for the Fedora releases).

@btashton Have you been able to make any progress on this? If not, then we may just close this issue and add Fedora support to our own repository.

[btashton]

@rossmcdonald I was waiting for the new storage backend to be stable before I picked this up again. I think most of the deps I have RPMs for in my copr. If you guys think it is stable enough for me to build a package for Fedora/EPEL I am happy to look at it again. The RPM that I have internally is deployed on our servers.

[beckettsean]

We are experimenting with simplifying our open GitHub Issues. This feature request has been rolled into an aggregate issue for all packaging requests, so that we can close this issue until we are ready to work on it.

You may continue to make comments here. Closing the issue does not mean we are rejecting this idea.

[ILMostro]

@btashton Just to clarify, was your work completed for fedora in the copr repo? I see a couple of successful builds for epel only from 3 years ago. I'm not sure if the epel version can/should be used on fedora or if it's a no-go due to the fact that it has not been updated in 3 years.
Additionally, the tracking issue here shows that this work was never completed. I appreciate any status update. Thank you.

[btashton]

@ILMostro What is in the copr repo is old and does not represent the best way forward. Fedora has relaxed the bundled vendor package rules a bit since then which will make packaging it much easier. I have not used InfluxDB in several years, but let me know if there is something I can do to help.

[ILMostro]

Thanks for the update. As of right now, the upstream pre-built rpm seems to work on Fedora28. That should suffice. Thanks again, @btashton

https://dl.influxdata.com/influxdb/releases/influxdb-1.6.2.x86_64.rpm