Document the existence of the Debian repository

[jameshfisher]

The README describes how I can build and then package logstash-forwarder. However, it would be much more convenient if there were either an archive of deb files, or a Debian/Ubuntu repository. Then I wouldn't have to bother with Ruby, FPM, Go, and any other transient build-time dependencies.

[jameshfisher]

I can see things about logstashforwarder at http://packages.elasticsearch.org/ but I don't know how to interpret it.

[jamtur01]

I believe there are currently RPM and DEB packages but not officially published. From memory they are next on the list after the Logstash 1.4 release. /cc @electrical.

[jameshfisher]

Oh hi again @jamtur01 :-)

Okay. Yeah, it would be super helpful. I want to automatically provision my machines running logstash-forwarder, but doing

install go
install git
git clone tagged version
cd
go build
install gem
gem install fpm
make deb
remove fpm
remove gem
remove git
remove go
dpkg -i whatever deb file

is a right pain in the ass.

[electrical]

Hi,

We indeed setup a yum and deb repository for LSF.

For DEB add the following to your sources.list:

deb http://packages.elasticsearch.org/logstashforwarder/debian stable main

run apt-get update

then you should be able to install it via apt-get install logstash-forwarder

[jblezoray]

Hi electrical,
This procedure should definitely be documented on the README.md. Is there a good reason for it not to be ?

[electrical]

@jblezoray i think we haven't got around to do that yet i'm afraid :-(

[deviantony]

+1 this needs to be more visible !!

[alexandresalome]

Don't forget to add the GPG key:

curl -o - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
# or
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -

[jblezoray]

The repositories existence is documented on the elasticsearch blog.

[Routhinator]

@jblezoray that blog does not document this repo, only the logstash core, which is far too heavy for 90% of machines. The forwarder repo should be documented as well.

[Routhinator]

The deb package also feels strangely incomplete. There's no default conf file. Looking at the /etc/init.d/logstash-forwarder script, it appears it's looking for /etc/logstash-forwarder

Is that a file named logstash-forwarder, or a dirname where it expects to find logstash-forwarder.conf?

[Routhinator]

Ah, it's looking for a /etc/logstash-forwarder file.

[geerlingguy]

Additionally, for RedHat/CentOS, you can install via the official repo by adding the GPG key, then add the following to /etc/repos.d/logstashforwarder.repo:

[logstashforwarder]
name=logstashforwarder repository
baseurl=http://packages.elasticsearch.org/logstashforwarder/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Now you can yum install logstash-forwarder. Some of the other setup steps are a little annoying on RedHat (it looks like the official docs lean towards debian/ubuntu instructions), but at least you can use a package manager this way...

[jordansissel]

The logstash-forwarder yum/deb repos are still a work in progress. Today, the recommended way to build lsf is from git master, at least until we do another release.

[geerlingguy]

@jordansissel - Good to know, and that explains why documentation is nonexistent :)

Having official packages would go along way towards easier/faster adoption, of course, but I know it takes time to get the repos set up and running efficiently.

[jordansissel]

The biggest blocker on adoption of lsf is presently difficulty users have in generating SSL certificates. We're working on that.

[martinb3]

Does the repo have regular releases? I see 0.3.1 in there, but according to github, that was released almost a year ago. Perhaps it's just LSF itself that isn't regularly doing version increments? I see a lot of merges in Github. I'm using config management tools to do this, and without a package, that work gets a lot uglier.

[zabbal]

Wow, this has to be in README: I've almost trashed the idea of using it because all the hassle with build it through gem-fpm-go-whatever described in readme is plain insane - it's completely unmaintainable.

[ljsommer]

Yeah this really needs to have packages. This is kind of a mess.

[mahnunchik]

+1

[burdandrei]

and today, these repos disappeared =(

[jordansissel]

The repo was delivering know buggy version of lsf for months. I tried to
purge the files a few months ago and thought it was done, but I realized
yesterday that I forgot another place. Fixed it by removing the files.

We will be doing another release of lsf soon.

-Jordan

On Sunday, February 8, 2015, Andrei Burd notifications@github.com wrote:

and today, these repos disappeared =(

—
Reply to this email directly or view it on GitHub
#184 (comment)
.

[malclocke]

😞 just went to push a deploy that relied on this repo and now it's 404. I guess the official line would be not to use it even if it does come back, seeing as it is not publicised?

[jordansissel]

@malclocke Mentioned previously, the recommendation is to build from master. The package previously known as "logstash-forwarder 0.3.1" was very buggy and caused trouble for so many new users that I purged it.

[topperge]

Purging the repo also caused the "official" puppet module to break as well
https://github.com/elasticsearch/puppet-logstashforwarder/blob/master/manifests/repo.pp#L35

[tomsommer]

You could also just push 0.3.2 to the repo instead of purging everything? Breaking yum/apt for everyone...

[deviantony]

Also please note that I cannot build the package using make deb on Ubuntu 14.04.

$:~/logstash-forwarder$ make deb
pleaserun --install --no-install-actions --install-prefix ./build \
        --overwrite -p sysv -v lsb-3.1 /opt/logstash-forwarder/bin/logstash-forwarder 
make: pleaserun: Command not found
make: *** [generate-init-script] Error 127

And if I try to install pleaserun :

$ sudo gem install pleaserun
ERROR:  Error installing pleaserun:
    mustache requires Ruby version ~> 2.0.

This sucks a lot.

[amosshapira]

@deviantony +1 for not being able to compile the package myself, I just hit the same problems myself.

I'm now stuck without a way to rebuild our Vagrant Basebox. I'll have to somehow recover the package from an existing box and setup our own repo mirror (I'm aware that this is supposed to be a buggy version but it works for us and I don't see a reasonable alternative, this is a far deeper rabbit hole than I intended to dig).

[PSiAU]

You can grab the package from the dropbox account belonging to @burdandrei (thank you!). Look at the comments above to find it.
What I ended up doing was installing the 0.3.1 package, then substituting the logstash-forwarder binary with my own built version from the master branch. That way the logstash-forwarder install is semi managed but running the latest binary. Like many others I had problems with mustache requiring Ruby 2.0 (Debian Wheezy comes with 1.8) which prevented me from building my own package hence why I decided on this route.

[RBLU]

+1

this small issue breaks basically all our ansible playbooks because "aptitude update" fails with 404, so I now need to remove this repo from the sources.list of all machines before I can install any updates again. Will do that now, but would also be very grateful for a new repo we can point to.
Thanks for your good work anyway.

[electrical]

Hi all,

We are working hard on getting LSF ready for a new release.
Once that's done we will re-activate the repo's again.

[JDiPierro]

Thanks for the update electrical. Just found this today when deploying a new server and installing logstash-forwarder failed :( Looking forward to the new release.

[astromoose]

@electrical looking forward to it 👍

[pnegahdar]

@electrical thanks for the update. Any eta on reactivating the repo? Our builds relied on it.

[DukeZeppelin]

think whould really good if appear possibility to choose default path for config file
And leave this file editable before making "make rpm"
it really whould be great!

[Siljanovski]

Hi guys, any news, eta or something ?

[adaniko]

This one bit me hard yesterday.

The clear take away for a user is to mirror everything that's mission critical, if you aren't already.

@jordansissel and all other contributors: Thank you for your work on this and for providing a repo. It's a huge help in getting it (and logstash in general) deployed.

My two cents, in the hope of providing some constructive criticism:

If we want people to treat this as a critical part of their infrastructure (and thus integrate it into their deployment scheme), being able to count on the deploy process is incredibly important. In the future, especially with repositories or anything that is a potential dependency for deployment, it would be a huge help to make sure that changes are either: a) non-breaking or b) publicly notified far in advance of a breaking change.

Just my opinion, but I think that creating a new repository would have been a much better option based on what we know so far about the reasons for the change (though I acknowledge that there may be factors that have not come to light yet). After establishing a working, opt-in alternative and providing a reasonable notification period, it would be okay to kill the current repo, redirect it, or replace it.

Taking a very considered approach to breaking changes is important in being a steward of any public-facing software. I think the fact that this change affected so many people testifies to the fact that logstash-forwarder and the people contributing to it are doing important (and hopefully appreciated) work. With some additional process in place, this should be something that can be avoided in the future.

It'd be nice, once a fix is in place, to do a post-mortem so that there are some concrete steps that people can plan on in the future.

I think we all learned a lot, hopefully with out too much long term pain.

Thanks again for the work you're doing.

[TimmGr]

Hallo everybody,

the last couple of weeks I was busy setting up a Logging Environment at work. It is a feasibility study on how to avoid splunk.
I was following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04
Note that I am no serveradmin by profession and consequently it took me a considerable amount of time.

Just as I am about to tell the team, that we can go with the logstash-graylog combo, the repository goes offline. Well, that was a little bit embarrassing for me :)

But I do not want to be the one who is complaining: Thank you for your hard work. Your software is cool. And will be even cooler, as soon as you make the installation fool-proof again.

Cheers

Oh one more thing: While installing it on Windows i spend a lot of time on the question on how to configure the paths (you have to use double ). Finally I found the answer, well hidden on:
https://logstash.jira.com/browse/LOGSTASH-1856
Maybe you want to present that also in the documentation. In case that is common knowledge please ignore this.

[zabbal]

If you can't maintain it I think you could simply ask the Debian Developers / Ubuntu MOTU teams to try to get an official maintainer for the DEB packages.

That would require fixing #299 with the help of lintian tool.

[mattwilliamson]

Dude. Killed so many deployments.

[brendangibat]

+1

[jordansissel]

#285 tracks the package repository.

[JDiPierro]

#385 actually.

[missteek]

wtf?? still no package and don't want to build. Going to use Fluentd instead!

[mahnunchik]

Can anyone in this thread provide a list of alternatives to logstash-forwarder with a more sensible approach to releases?

[Siljanovski]

Gents, I know this is not a long-term solution but it helped me a lot:
Here you can find one .sh file (I am not sys-admin so my bash scripting is basic).
https://github.com/Siljanovski/elasticsearch/blob/master/logstash-forwarder-installer.sh

I used this for installing manually, but with little modification you can put this into Chef or any other config management tool that you use.
I put both daemon script file and deb file for 64 bits into my repo since they were gone from other repos that I was using before. I will not remove them :)

The only thing that you need to do is to give access to your servers so they can scp the crt file and that's it. It works just after executing this script.

This might help you a bit.

[driskell]

@mahnunchik I maintain an alternative called Log Courier - it doesn't have Deb or rpm at moment but will have CentOS/RedHat rpm soon. Not had any demand for Deb as such.

[deviantony]

@driskell 's alternative is great but it also lack the presence of a debian repository, which is mainly why I didn't switch to it.

[pnegahdar]

@Siljanovski yup, pretty much what I'm doing. Careful with the the wget on master you should use "https://github.com/elasticsearch/logstash-forwarder/archive/v0.3.1.tar.gz" instead. Also the touch should be unnecessary with the echo > creates if DNE.

[amosshapira]

What I ended up doing for now is to salvage a copy of the .deb file from the cache of one of our running servers and build a repository around it on S3 using reprepro. Eventually I'll get around to mirror all such "small" external repositories we use, and would recommend everyone else to do the same.

If you need a copy of the repository then let me know how you want me to get it to you.