Binary packaging for distros

[obfuscurity]

The project has been pretty slack in the past about making sure the binary packages work properly and are supported across the most popular versions. I don't think we need to go crazy, but it would be nice to set some goals, achieve them in master, and then backport the necessary bits for 0.9.x.

Here's a shopping list of the distro versions I'd like to see us support. If I'm crazy or there are dependency issues (i.e. Python or Django), let's talk about that here.

• Fedora 20
• CentOS 5
• CentOS 6
• CentOS 7
• Ubuntu 14.04 LTS
• Ubuntu 12.04 LTS
• Debian 6
• Debian 7

/cc @thorrsson @bmhatfield @robbkidd @mzupan @mckern @rwoolley, who have all volunteered to help with packaging or have worked on Graphite packaging in the past.

/cc @brutasse @SEJeff @bitprophet

P.S. This issue is currently only open in graphite-web, but obviously we'll need to consider carbon, whisper and ceres as well.

[mckern]

First read through, those don't look crazy. That list is really only supporting two platforms, with minor variations among the flavors of those platforms (init scripts, dependency package names, versions on older releases like EL5 and Ubuntu 12.04 LTS). We'd need to work out the dependencies, what's provided by the vendor already, what's provided but too old to use, and then how to resolve that. I haven't looked at any existing binary packages in about 12-18 months but I can do that this weekend. Any documentation about how they're built right now?

[obfuscurity]

I think the only bits (at least in graphite-web) currently involved in packaging are setup.py, setup.cfg, and then a postinstall script at distro/redhat/misc/postinstall.

[obfuscurity]

P.S. I think that @thorrsson has started to dig into the rpm stuff already, so please be sure to sync up before duplicating efforts.

[deniszh]

I can say that support 0.9.13 in Ubuntu 12.04 and Debian 6 will be quite hard. AFAIK Ubuntu 12.04 has only Django 1.3 - http://packages.ubuntu.com/precise/python-django and Debian 6 has Django 1.2 - https://packages.debian.org/squeeze/python-django and 0.9.13 requires Django 1.4
And AFAIR I faced with many other issues with python packages even on 0.9.10.

[mzupan]

Do we want to have the files live in the same directories across platform or stick to debian/ubuntu standards vs redhat standards. Sometimes they don't differ much but mainly configs they differ.

[robbkidd]

I'm using 0.9.12 on Ubuntu 12.04 by judicious application of hw-cookbooks/graphite. Would it be crazy to go down the path of an omnibus package install of graphite and all its dependencies? I think most of what the Heavy Water graphite cookbook installs by way of python packages could be done in an embedded python in an omnibus install.

[deniszh]

IMHO https://github.com/hw-cookbooks/graphite is good candidate for filling 'Install by Chef' docs section. Binary package implies installation of some amount deb or rpm packages and that's all. You can transfer these packages to own repo and install it even without Internet access - and it must work after that.

[mckern]

👍 to not building omnibus packages if it can be avoided. It'll inflate packages considerably (where do you dry the line in the stack?) and it'll increase complexity. @deniszh Have you already started digging into deb packages?

[josephholsten]

I'd prefer to provide the debs via a launchpad ppa. 90% of the challenge there is just following debian packaging guidelines. I'm already acquainted with debhelper and dh-golang, and I think just grokking dh_python(1) should be enough to get us where we want to be.

For the debian stack, I recommend we focus on Ubuntu 14.04 first and try to get into Utopic (14.10). Then we can figure out a backporting strategy for Ubuntu 12.04, Debian 7 & Debian 6.

Do we want to support both apache and nginx webservers? mod_python? uwsgi? gunicorn? I've personally liked boring apache+mod_python, but I understand that's not the coolest tech.

For carbon, we'll probably want to support sysv-init, upstart & systemd. It'll almost certainly need tweaked ulimits . Otherwise they should be relatively boring.

[obfuscurity]

I would also prefer to avoid omnibus packages, as well as using external cookbooks as our build specification. Anything we release should be managed in-tree imho.

[mckern]

I think we should start w/ HTTPD + mod_python if it's the simplest supportable configuration; it doesn't need to be cool. It's easier to pick one target and increment than to try to shotgun them all. It'll also make reusing logic a little easier, so that even though we can't reuse post/pre scripts we can make sure that the packages do the same things on different platforms.

[mckern]

Oh, and (unfortunately) 👍 to supporting sysv-init, upstart, and systemd. It's a pain, but it's an upfront pain.

Quick status check for the names that have chimed in, who's working on what specifically already? Would like to avoid duplicating effort or competing implementations if we can suss out who will tackle what.

[josephholsten]

Oh, I'm working on debhelperizing for ubuntu 14.04. I'll be using apache httpd & mod_python.

[obfuscurity]

It sounds like @thorrsson and @mckern should sync up on rpm work and @josephholsten is taking lead on debs. Anyone else that wants to pitch in, please do, but touch base with those gentlemen first.

Thanks everyone! 💖

P.S. I would prefer that we target apache + mod_wsgi over mod_python unless someone has a good reason not to. Our own example vhost config uses mod_wsgi and our documentation covers mod_wsgi for virtualenv installs. We currently have no documentation (other than a passing mention) for mod_python; I have no real arguments against standardizing our packages on it, but we should also add an example config and documentation for mod_python if we wish to go down that route.

[robbkidd]

/me discards omnibus idea
/me starts reading up on debhelper and dh-python

(Update: dh_python deprecated in 12.04+, apparently. Directs to using dh_python2)

[mckern]

HTTPD + mod_wsgi sounds like a plan. @thorrsson, how's things on your end?

[josephholsten]

got it, mod_wsgi it is.

[josephholsten]

I'm working on the debian packaging in #865. My debian-specific notes will be going there.

[thorrsson]

Hey folks,
my $0.02 for the comments from today:

• I have boxes initialized for testing on OEL6.5, Centos (6.5, 7),
  Fedora 20 and RHEL6.5 (I can probably initialize a EL 5 box fairly quickly
  too)
• I don't think that we need to get into the omnibus game, the existing
  cookbooks are solid and fairly easy to wrap (as are the puppet modules)
• mod_python should not be an option, there was a noticeable performance
  difference compared to mod_wsgi, I would stick to wsgi
• I wouldn't mind including optional nginx/uwsgi configs as bonus
  points, though I am not sure if we should simply express a dependency on
  the webserver and provide example configs or actually manage the site
  configs and the webserver. Whenever I deployed graphite at $previous_jobs,
  The webserver was always managed by my Configuration Management As well,
  If we handle the webserver and the config we may need to account for shared
  server situations where folks don't run graphite exclusively on the box. I
  have also seen a ton of different existing apache configs (sites-* vs /conf
  vs hacking the httpd.conf etc) that we may not want to try and work around

Hi @mckern! This is where I am at as of last night:
I am building on Centos 6.5 for the time being, though spent a good part of
yesterday evening getting an environment that works (thanks to just
replacing my laptop :) )
I have started with Carbon, based off of master as well as ensuring I can
port back to 0.9.x
I put in a PR last night that Jason merged to fix the install-lib location
that was broken (it dumped carbon libs into ./webapp), and we will need to
make a decision on graphite-project/carbon#148
regarding managing the installation of the init scripts or not (I hacked
this into my local fork in order to build on Centos, and determines if we
need to worry about init script installation vs just creating the scripts)

I can produce an RPM that is targeted at python 2.7 and manages Carbon and
the init scripts using bdist_rpm at this point. I also generated the .spec
while I was at it and found that we needed to add the long_description
(which was also merged today). I also tried out building with FPM, which
had some quirks that I didn't bother working around. The artifact installs
cleanly, with a dependency on whisper (which I built out really quickly
just to get carbon installed) The services all start as expected as well.

On Sat, Aug 23, 2014 at 5:24 PM, Ryan McKern notifications@github.com
wrote:

HTTPD + mod_wsgi wounds like a plan. @thorrsson
https://github.com/thorrsson, how's things on your end?

—
Reply to this email directly or view it on GitHub
#864 (comment)
.

[deniszh]

Currently I'm installing Graphite on my prod envs using deb packages built
by FPM. Is it good idea or it's better to stick with standard Debian build
system?
+1 for Apache + mod_wsgi
Also maybe we could use supervisord for starting daemons? It's pythonish
and cross-platform.
24 авг. 2014 г. 1:14 пользователь "Ryan McKern" notifications@github.com
написал:

[image: 👍] to not building omnibus packages if it can be avoided.
It'll inflate packages considerably (where do you dry the line in the
stack?) and it'll increase complexity. @deniszh
https://github.com/deniszh Have you already started digging into deb
packages?

—
Reply to this email directly or view it on GitHub
#864 (comment)
.

[obfuscurity]

It seems to me that having a real build spec for debs would be a good thing. But to be honest, I haven't followed the development of fpm closely to know how well a job it does at porting one format to another.

[bmhatfield]

As someone who has previously deployed Graphite via packages (rpms, then debs, now straight python packages installed by pip), I think this thread is headed in the right direction.

Some thoughts:

• I have previously had success using "python to OS package" plugins to setup.py, bdist_rpm is built in, and bdist_deb is a short install away: https://pypi.python.org/pypi/stdeb
• I think it's probably wise to try to "stick to the distro" in terms of init support. Don't switch or add supervisors on people (eg, no runit, no supervisord, bluepill, etc). This means likely writing a few init scripts. Word to the wise: carbon-cache and carbon-relay are not supported by upstart without 'hacks' such as forcing them to run in the foreground, which affects where logs are written. It might be worth doing some work on their daemonization approach in this regard.
• I think it's also probably wise to "stick to the distro" in terms of Python support. I think this will be the most controversial, as I found that people sometimes want to virtualenv graphite, etc. This also means that we'd have to be very clear about which distros are supported (as @obfuscurity already laid out!). It would also indirectly prescribe which Python versions must be supported, which may or may not be something we want.
• Carbon was the most difficult to package in my opinion, due to making sure the config files are sane. Currently Graphite generally assumes deployment into /opt/graphite - will we change that up to be more "FHS"-like? (You can make an argument that /opt is fine, but I think ya'll see where I'm going with that).
• Graphite web is the second hardest. I see some discussion above about choosing to deploy under a webserver (eg, apache, nginx, whatever) or specifying a dependency. My current thinking is specify a chosen webserver as an optional dependency (eg, apache), and let the admin choose. Though this is quite different than where we are at today, it's probably wise to provide a python-native django app server if they don't choose anything, detectable by postinstall.
• Speaking of Django and other 'probably not available' python libraries: If we're going to host this via ppa (or s3 http apt repo, which is also quite doable), I'd recommend (though not excitedly) either cloning or packaging each dependency of Graphite into it's own OS package hosted in the PPA as well.
• Because the above is a ton of work: Is it worth building "dependencies included" packages of Graphite instead of separating out each Python library dependency? In the Debian world, that's a no-no, but like the way some people vendor Bottle when building a Bottle app, it might make sense to vendor all of the dependencies of graphite-web into the graphite-web package, rather than depending on them to be installed separately. It'll make the package larger (though not omnibus large), but it will dramatically simplify deployment and upgrades in the future. My gut instinct is that this is what people "probably want" anyways - especially if said dependencies only appear to Graphite-web/whatever.
• Whisper is quite easy to package.
• What will the support be for Ceres and such?

[gwaldo]

@deniszh The hw-cookbooks/graphite cookbook is what I intend on using as the recommendation for the 'install-via-chef' section of the docs, though this is heavily caveated.

First, is that last I checked a week or two ago, they are looking to cut a new release that is much more library/provider oriented than the current version.

Second, is that if we get the packaging more sane, that could simplify the cookbooks needed overall.

[bmhatfield]

So, now that I've dropped a bunch of random thoughts, let me approach this a different way.

Pre-caveat: I believe strongly in OS packages wherever applicable. I don't love the proliferation of every language gets it's own package manager.

So here goes: What problem are we trying to solve by packaging Graphite into OS packages? As is probably obvious from my above comment, I used to invest a lot of effort into packaging Graphite. But I also invested an equal amount of effort into making sure my configuration manager understood what the package was doing so it could "correct" it for my deployment (eg, move /opt/graphite/storage/ to a larger partition). Each thing the package did in a roughly 'standards' way, I had to go back and tweak to make things work better for my particular deployment. Every time Graphite needed to be upgraded, I was amazed at the amount of changes that went into both packaging and configuring it vs. the previous version.

Ultimately, I ended up in a place where I just mirror the pypi tarballs and install them using pip, then perform my configuration tasks as needed. Somehow, it ends up feeling simpler, though I agree it's a lot of work repeated community wide.

Anyways, I think it's worth being clear what goals we are trying to achieve by creating 'the way' of installing Graphite; to make sure that we actually achieve them. Packaging won't in and of itself make deploying Graphite meaningfully easier, given that much of the challenge of installing it comes after the bits are actually installed onto the machine.

EDIT: PS, to make sure this is super clear: I am pro-binary packages for Graphite. But it'd be a good idea to align on what problems we can solve, vs which we cannot. We can likely solve "make getting the bits on the machine" more like what Admins are used to w/r/t to 'sudo aptitude install graphite-web' or something, but probably not make it particularly easier to install Graphite.

[obfuscurity]

On Sun, Aug 24, 2014 at 09:23:17AM -0700, Brian Hatfield wrote:

So, now that I've dropped a bunch of random thoughts, let me approach this a different way.

Pre-caveat: I believe strongly in OS packages wherever applicable. I don't love the proliferation of every language gets it's own package manager.

So here goes: What problem are we trying to solve by packaging Graphite into OS packages? As is probably obvious from my above comment, I used to invest a lot of effort into packaging Graphite. But I also invested an equal amount of effort into making sure my configuration manager understood what the package was doing so it could "correct" it for my deployment (eg, move /opt/graphite/storage/ to a larger partition). Each thing the package did in a roughly 'standards' way, I had to go back and tweak to make things work better for my particular deployment. Every time Graphite needed to be upgraded, I was amazed at the amount of changes that went into both packaging and configuring it vs. the previous version.

IMHO we're trying to solve the problem of installing Graphite for users
who are inexperienced with Django and/or Python package/environment
tooling (pip, virtualenv, etc).

Ultimately, I ended up in a place where I just mirror the pypi tarballs and install them using pip, then perform my configuration tasks as needed. Somehow, it ends up feeling simpler, though I agree it's a lot of work repeated community wide.

Anyways, I think it's worth being clear what goals we are trying to achieve by creating 'the way' of installing Graphite; to make sure that we actually achieve them. Packaging won't in and of itself make deploying Graphite meaningfully easier, given that much of the challenge of installing it comes after the bits are actually installed onto the machine.

I don't expect anything beyond this, i.e. we should not lean on our
packaging to handle complex configurations (either vertical or
horizontal scaling).

Jason Dixon
http://obfuscurity.com/
https://twitter.com/obfuscurity

[brutasse]

Graphite is popular enough that it's already provided in official distro packages (at least for debian and ubuntu). IMO, if we want to provide distro package files we have to work with debian / ubuntu / centos people to write packaging scripts they're happy with directly in our repos. This way distro packages and user-built packages will look the same and it'll be easer for people to switch from an official package to a custom one. So, can we get in touch whith Debian / Ubuntu / Centos developers to have them bless the results of what we're doing? Or look at what they've done in official repos and start from there? We need to get to a point where distro developers don't have to maintain packaging patches. Also let's hear what they say about supporting older OS versions than current stable/LTS… I'm not sure 12.04 / debian 6 is doable, might have to check.

(and if they're all in favor of dropping the /opt/graphite prefix, let's do it asap)

What I did for graphite-api is an omnibus package (fpm) that provides all dependencies, completely isolated from system packages and with an init script that starts a service with gunicorn. I left apache/nginx configuration instructions in the docs. Debian people seem to be interested in having it packaged in sid+backports, in which case I'll merge whatever pull request they make to get a debian/ directory they're happy with. (and they probably won't go the omnibus way :)

[deniszh]

@brutasse, only 14.04 and 14.10 ubuntu has current graphite packages. RHEL / CentOS / Fedora / Debian 6 / Debian 7 / Ubuntu 12.04 are not. And yep, in Ubuntu there's no /opt/graphite prefix, all packages are installing to appropriate (by distro) places. /etc/graphite/local_settings.py looks odd to me, but good from distro perspective.

[mckern]

@thorrsson Have you got this work in a branch somewhere? And have you got the .spec files you've created available anywhere?

[vincentbernat]

Current distro packaging for Debian/Ubuntu can be found here:
http://sources.debian.net/src/graphite-web/latest/debian/
http://sources.debian.net/src/graphite-carbon/latest/debian/

[mckern]

Here's something that Fedora is cooking up:
http://pkgs.fedoraproject.org/cgit/graphite-web.git/tree/

[thorrsson]

Hey folks,
I have had some personal life happen and am probably not going to have any
spare cycles to work on this. Sorry to bail out, feel free to keep me on
the thread and I can provide input and possibly review if time permits.
Cheers
Tim

On Tue, Aug 26, 2014 at 10:06 AM, Ryan McKern notifications@github.com
wrote:

Here's something that Fedora is cooking up:
http://pkgs.fedoraproject.org/cgit/graphite-web.git/tree/

—
Reply to this email directly or view it on GitHub
#864 (comment)
.

[mckern]

Taking a look at the Fedora RPM spec files today, and experimenting to see if they're of use/benefit here. Will try building across all the requested RPM platforms, and will report back.

[obfuscurity]

🔥

[mckern]

@obfuscurity thoughts on the use in Fedora of the python- prefix for carbon & whisper?

[obfuscurity]

@mckern I've never been fond of the python- prefixing. Is there a reason for it?

[mckern]

Usually denotes that something is specifically going to drop libraries into your python libpath 9r requires python to be of any use (much like the ruby- and rubygem- prefixes). Here's the relevant details from Fedora.

[obfuscurity]

Can we reconcile between Fedora's prefix requirements and a clean upgrade path for existing RPM users?

[mckern]

Where can I find some existing non-Fedora RPMs or spec files? Using metadata like Requires or Obsoletes, we could craft packages that simply replace existing packages as part of a known upgrade but I'd need to know what sort of metadata the existing packages provide.

Or am I misunderstanding, and you're proposing that we make it possible to upgrade from Fedora/EPEL graphite/carbon/whisper packages to ones provided first-hand by the Graphite project?

[mckern]

@obfuscurity Any followup to the outstanding questions ^^ ?

[obfuscurity]

@mckern Hmm, I don't know about the specs. I would have assumed that someone on this thread might know where to find them. @bmhatfield perhaps?

I would love to bring in the official distro maintainers and get their $0.02 on this effort. I worry now that this is seen as duplication of their work, but I'd still rather have an officially maintained package hooked into our build (and tests /cc @jssjr) than force users to wait on the upstream maintenance cycles.

[mckern]

I may have misunderstood; I was working from the assumption that there were already pre-existing but less-than-immaculate packages.

[obfuscurity]

Can anyone take a look at graphite-project/carbon#355 with me?

[damm]

Any reason you can't Omnibus Graphite? there's only really a few targets and trying to address many versions will only complicate life.

[SEJeff]

Not a fan of omnibus, I'd rather be a good upstream and work with distributions to get our software included in their repos, and make it easy for our users to install on the various distros with somewhat reasonable native support. Omnibus encourages curl ...| sh, which makes me a very sad Keanu.

[sebito91]

I'm actually a member of the fedora-admin community and have been touting
Graphite and collectd for Fedora 22. We are currently working to add
graphite to the existing metrics collection backend (currently using
RRDTool) so I'm happy to help with that arm.

On Monday, January 12, 2015, Jeff Schroeder notifications@github.com
wrote:

Not a fan of omnibus, I'd rather be a good upstream and work with
distributions to get our software included in their repos, and make it easy
for our users to install on the various distros with somewhat reasonable
native support. Omnibus encourages curl ...| sh, which makes me a very sad
Keanu.

—
Reply to this email directly or view it on GitHub
#864 (comment)
.

[obfuscurity]

@sebito91 That would be great, and now's the time to pitch in. Per graphite-project/carbon#355, it seems that our build tarball is missing some distro stuff. Admittedly, I built it on a Mac, but I'm surprised that would matter. We can't very well make multiple tarballs available, one per distro/platform, for a single release (for PyPI).

[sebito91]

Happy
      to start looking at the issue.

      Quick update, mentioned pacakaging graphite-web for release
      during the Fedora-admin meeting this week and the team is
      stoked. They're keen to start testing it out on the
      infrastructure servers as soon as we can get it ready!
  Sebastian Borza

---

PGP: EDC2 BF61 4B91 14F2 AAB4 06C9 3744 7F3F E411 0D3E
On 1/12/15 11:25 PM, Jason Dixon wrote:

  @sebito91 That would be great, and
    now's the time to pitch in. Per graphite-project/carbon#355,
    it seems that our build tarball is missing some distro stuff.
    Admittedly, I built it on a Mac, but I'm surprised that would
    matter. We can't very well make multiple tarballs available, one
    per distro/platform, for a single release (for PyPI).
  —
    Reply to this email directly or view
      it on GitHub.

[obfuscurity]

@sebito91 Awesome.

[ArturGajowy]

Any chance to have 1.0.2 RPMs for centos anytime soon?

[piotr1212]

@ArturGajowy Do you mean 1.0.2 in EPEL 7 repository? For that to work Django will need to be updated first.

[ArturGajowy]

Does this mean "just forget it, the python world uses pip"?

[piotr1212]

@ArturGajowy I'm still not really sure which packages you are referring to.

[ArturGajowy]

graphite-web, carbon, whisper

I tried installing from pip, but got this:

[piotr1212]

I tried installing from pip, but got this:

I'm quite sure pip does not install in /usr/lib/python2.7/site-packages/graphite by default. Make sure to remove old installations.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.