Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document the existence of the Debian repository #184

Closed
jameshfisher opened this Issue Mar 30, 2014 · 72 comments

Comments

Projects
None yet
@jameshfisher
Copy link

jameshfisher commented Mar 30, 2014

The README describes how I can build and then package logstash-forwarder. However, it would be much more convenient if there were either an archive of deb files, or a Debian/Ubuntu repository. Then I wouldn't have to bother with Ruby, FPM, Go, and any other transient build-time dependencies.

@jameshfisher

This comment has been minimized.

Copy link
Author

jameshfisher commented Mar 30, 2014

I can see things about logstashforwarder at http://packages.elasticsearch.org/ but I don't know how to interpret it.

@jamtur01

This comment has been minimized.

Copy link
Contributor

jamtur01 commented Mar 30, 2014

I believe there are currently RPM and DEB packages but not officially published. From memory they are next on the list after the Logstash 1.4 release. /cc @electrical.

@jameshfisher

This comment has been minimized.

Copy link
Author

jameshfisher commented Mar 30, 2014

Oh hi again @jamtur01 :-)

Okay. Yeah, it would be super helpful. I want to automatically provision my machines running logstash-forwarder, but doing

install go
install git
git clone tagged version
cd
go build
install gem
gem install fpm
make deb
remove fpm
remove gem
remove git
remove go
dpkg -i whatever deb file

is a right pain in the ass.

@electrical

This comment has been minimized.

Copy link

electrical commented Mar 30, 2014

Hi,

We indeed setup a yum and deb repository for LSF.

For DEB add the following to your sources.list:

deb http://packages.elasticsearch.org/logstashforwarder/debian stable main

run apt-get update

then you should be able to install it via apt-get install logstash-forwarder

@jameshfisher jameshfisher changed the title Debian repository or archive of .deb files Document the existence of the Debian repository Mar 31, 2014

@jblezoray

This comment has been minimized.

Copy link

jblezoray commented Jun 17, 2014

Hi electrical,
This procedure should definitely be documented on the README.md. Is there a good reason for it not to be ?

@electrical

This comment has been minimized.

Copy link

electrical commented Jun 17, 2014

@jblezoray i think we haven't got around to do that yet i'm afraid :-(

@deviantony

This comment has been minimized.

Copy link

deviantony commented Jun 17, 2014

+1 this needs to be more visible !!

@alexandresalome

This comment has been minimized.

Copy link

alexandresalome commented Jul 3, 2014

Don't forget to add the GPG key:

curl -o - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
# or
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
@jblezoray

This comment has been minimized.

Copy link

jblezoray commented Jul 15, 2014

The repositories existence is documented on the elasticsearch blog.

@Routhinator

This comment has been minimized.

Copy link

Routhinator commented Aug 13, 2014

@jblezoray that blog does not document this repo, only the logstash core, which is far too heavy for 90% of machines. The forwarder repo should be documented as well.

@Routhinator

This comment has been minimized.

Copy link

Routhinator commented Aug 13, 2014

The deb package also feels strangely incomplete. There's no default conf file. Looking at the /etc/init.d/logstash-forwarder script, it appears it's looking for /etc/logstash-forwarder

Is that a file named logstash-forwarder, or a dirname where it expects to find logstash-forwarder.conf?

@Routhinator

This comment has been minimized.

Copy link

Routhinator commented Aug 13, 2014

Ah, it's looking for a /etc/logstash-forwarder file.

@geerlingguy

This comment has been minimized.

Copy link

geerlingguy commented Aug 26, 2014

Additionally, for RedHat/CentOS, you can install via the official repo by adding the GPG key, then add the following to /etc/repos.d/logstashforwarder.repo:

[logstashforwarder]
name=logstashforwarder repository
baseurl=http://packages.elasticsearch.org/logstashforwarder/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Now you can yum install logstash-forwarder. Some of the other setup steps are a little annoying on RedHat (it looks like the official docs lean towards debian/ubuntu instructions), but at least you can use a package manager this way...

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Aug 26, 2014

The logstash-forwarder yum/deb repos are still a work in progress. Today, the recommended way to build lsf is from git master, at least until we do another release.

@geerlingguy

This comment has been minimized.

Copy link

geerlingguy commented Aug 26, 2014

@jordansissel - Good to know, and that explains why documentation is nonexistent :)

Having official packages would go along way towards easier/faster adoption, of course, but I know it takes time to get the repos set up and running efficiently.

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Aug 26, 2014

The biggest blocker on adoption of lsf is presently difficulty users have in generating SSL certificates. We're working on that.

@martinb3

This comment has been minimized.

Copy link

martinb3 commented Sep 26, 2014

Does the repo have regular releases? I see 0.3.1 in there, but according to github, that was released almost a year ago. Perhaps it's just LSF itself that isn't regularly doing version increments? I see a lot of merges in Github. I'm using config management tools to do this, and without a package, that work gets a lot uglier.

@zabbal

This comment has been minimized.

Copy link

zabbal commented Oct 31, 2014

Wow, this has to be in README: I've almost trashed the idea of using it because all the hassle with build it through gem-fpm-go-whatever described in readme is plain insane - it's completely unmaintainable.

@ljsommer

This comment has been minimized.

Copy link

ljsommer commented Nov 19, 2014

Yeah this really needs to have packages. This is kind of a mess.

@mahnunchik

This comment has been minimized.

Copy link

mahnunchik commented Dec 4, 2014

+1

@burdandrei

This comment has been minimized.

Copy link

burdandrei commented Feb 8, 2015

and today, these repos disappeared =(

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Feb 8, 2015

The repo was delivering know buggy version of lsf for months. I tried to
purge the files a few months ago and thought it was done, but I realized
yesterday that I forgot another place. Fixed it by removing the files.

We will be doing another release of lsf soon.

-Jordan

On Sunday, February 8, 2015, Andrei Burd notifications@github.com wrote:

and today, these repos disappeared =(


Reply to this email directly or view it on GitHub
#184 (comment)
.

@malclocke

This comment has been minimized.

Copy link

malclocke commented Feb 8, 2015

😞 just went to push a deploy that relied on this repo and now it's 404. I guess the official line would be not to use it even if it does come back, seeing as it is not publicised?

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Feb 8, 2015

@malclocke Mentioned previously, the recommendation is to build from master. The package previously known as "logstash-forwarder 0.3.1" was very buggy and caused trouble for so many new users that I purged it.

@topperge

This comment has been minimized.

Copy link

topperge commented Feb 10, 2015

Purging the repo also caused the "official" puppet module to break as well
https://github.com/elasticsearch/puppet-logstashforwarder/blob/master/manifests/repo.pp#L35

@tomsommer

This comment has been minimized.

Copy link

tomsommer commented Feb 10, 2015

You could also just push 0.3.2 to the repo instead of purging everything? Breaking yum/apt for everyone...

@deviantony

This comment has been minimized.

Copy link

deviantony commented Feb 16, 2015

Also please note that I cannot build the package using make deb on Ubuntu 14.04.

$:~/logstash-forwarder$ make deb
pleaserun --install --no-install-actions --install-prefix ./build \
        --overwrite -p sysv -v lsb-3.1 /opt/logstash-forwarder/bin/logstash-forwarder 
make: pleaserun: Command not found
make: *** [generate-init-script] Error 127

And if I try to install pleaserun :

$ sudo gem install pleaserun
ERROR:  Error installing pleaserun:
    mustache requires Ruby version ~> 2.0.

This sucks a lot.

@amosshapira

This comment has been minimized.

Copy link

amosshapira commented Feb 17, 2015

@deviantony +1 for not being able to compile the package myself, I just hit the same problems myself.

I'm now stuck without a way to rebuild our Vagrant Basebox. I'll have to somehow recover the package from an existing box and setup our own repo mirror (I'm aware that this is supposed to be a buggy version but it works for us and I don't see a reasonable alternative, this is a far deeper rabbit hole than I intended to dig).

@PSiAU

This comment has been minimized.

Copy link

PSiAU commented Feb 17, 2015

You can grab the package from the dropbox account belonging to @burdandrei (thank you!). Look at the comments above to find it.
What I ended up doing was installing the 0.3.1 package, then substituting the logstash-forwarder binary with my own built version from the master branch. That way the logstash-forwarder install is semi managed but running the latest binary. Like many others I had problems with mustache requiring Ruby 2.0 (Debian Wheezy comes with 1.8) which prevented me from building my own package hence why I decided on this route.

@RBLU

This comment has been minimized.

Copy link

RBLU commented Feb 17, 2015

+1

this small issue breaks basically all our ansible playbooks because "aptitude update" fails with 404, so I now need to remove this repo from the sources.list of all machines before I can install any updates again. Will do that now, but would also be very grateful for a new repo we can point to.
Thanks for your good work anyway.

@electrical

This comment has been minimized.

Copy link

electrical commented Feb 17, 2015

Hi all,

We are working hard on getting LSF ready for a new release.
Once that's done we will re-activate the repo's again.

@JDiPierro

This comment has been minimized.

Copy link

JDiPierro commented Feb 17, 2015

Thanks for the update electrical. Just found this today when deploying a new server and installing logstash-forwarder failed :( Looking forward to the new release.

@astromoose

This comment has been minimized.

Copy link

astromoose commented Feb 18, 2015

@electrical looking forward to it 👍

@pnegahdar

This comment has been minimized.

Copy link

pnegahdar commented Feb 18, 2015

@electrical thanks for the update. Any eta on reactivating the repo? Our builds relied on it.

@DukeZeppelin

This comment has been minimized.

Copy link

DukeZeppelin commented Feb 18, 2015

think whould really good if appear possibility to choose default path for config file
And leave this file editable before making "make rpm"
it really whould be great!

@Siljanovski

This comment has been minimized.

Copy link

Siljanovski commented Feb 20, 2015

Hi guys, any news, eta or something ?

@adaniko

This comment has been minimized.

Copy link

adaniko commented Feb 20, 2015

This one bit me hard yesterday.

The clear take away for a user is to mirror everything that's mission critical, if you aren't already.

@jordansissel and all other contributors: Thank you for your work on this and for providing a repo. It's a huge help in getting it (and logstash in general) deployed.

My two cents, in the hope of providing some constructive criticism:

If we want people to treat this as a critical part of their infrastructure (and thus integrate it into their deployment scheme), being able to count on the deploy process is incredibly important. In the future, especially with repositories or anything that is a potential dependency for deployment, it would be a huge help to make sure that changes are either: a) non-breaking or b) publicly notified far in advance of a breaking change.

Just my opinion, but I think that creating a new repository would have been a much better option based on what we know so far about the reasons for the change (though I acknowledge that there may be factors that have not come to light yet). After establishing a working, opt-in alternative and providing a reasonable notification period, it would be okay to kill the current repo, redirect it, or replace it.

Taking a very considered approach to breaking changes is important in being a steward of any public-facing software. I think the fact that this change affected so many people testifies to the fact that logstash-forwarder and the people contributing to it are doing important (and hopefully appreciated) work. With some additional process in place, this should be something that can be avoided in the future.

It'd be nice, once a fix is in place, to do a post-mortem so that there are some concrete steps that people can plan on in the future.

I think we all learned a lot, hopefully with out too much long term pain.

Thanks again for the work you're doing.

@TimmGr

This comment has been minimized.

Copy link

TimmGr commented Feb 23, 2015

Hallo everybody,

the last couple of weeks I was busy setting up a Logging Environment at work. It is a feasibility study on how to avoid splunk.
I was following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04
Note that I am no serveradmin by profession and consequently it took me a considerable amount of time.

Just as I am about to tell the team, that we can go with the logstash-graylog combo, the repository goes offline. Well, that was a little bit embarrassing for me :)

But I do not want to be the one who is complaining: Thank you for your hard work. Your software is cool. And will be even cooler, as soon as you make the installation fool-proof again.

Cheers

Oh one more thing: While installing it on Windows i spend a lot of time on the question on how to configure the paths (you have to use double ). Finally I found the answer, well hidden on:
https://logstash.jira.com/browse/LOGSTASH-1856
Maybe you want to present that also in the documentation. In case that is common knowledge please ignore this.

@zabbal

This comment has been minimized.

Copy link

zabbal commented Feb 23, 2015

If you can't maintain it I think you could simply ask the Debian Developers / Ubuntu MOTU teams to try to get an official maintainer for the DEB packages.

That would require fixing #299 with the help of lintian tool.

@mattwilliamson

This comment has been minimized.

Copy link

mattwilliamson commented Feb 23, 2015

Dude. Killed so many deployments.

@brendangibat

This comment has been minimized.

Copy link

brendangibat commented Feb 24, 2015

+1

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Feb 24, 2015

#285 tracks the package repository.

@JDiPierro

This comment has been minimized.

Copy link

JDiPierro commented Feb 24, 2015

#385 actually.

@missgonads

This comment has been minimized.

Copy link

missgonads commented Feb 25, 2015

wtf?? still no package and don't want to build. Going to use Fluentd instead!

@mahnunchik

This comment has been minimized.

Copy link

mahnunchik commented Feb 25, 2015

Can anyone in this thread provide a list of alternatives to logstash-forwarder with a more sensible approach to releases?

@Siljanovski

This comment has been minimized.

Copy link

Siljanovski commented Feb 25, 2015

Gents, I know this is not a long-term solution but it helped me a lot:
Here you can find one .sh file (I am not sys-admin so my bash scripting is basic).
https://github.com/Siljanovski/elasticsearch/blob/master/logstash-forwarder-installer.sh

I used this for installing manually, but with little modification you can put this into Chef or any other config management tool that you use.
I put both daemon script file and deb file for 64 bits into my repo since they were gone from other repos that I was using before. I will not remove them :)

The only thing that you need to do is to give access to your servers so they can scp the crt file and that's it. It works just after executing this script.

This might help you a bit.

@driskell

This comment has been minimized.

Copy link
Contributor

driskell commented Feb 25, 2015

@mahnunchik I maintain an alternative called Log Courier - it doesn't have Deb or rpm at moment but will have CentOS/RedHat rpm soon. Not had any demand for Deb as such.

@deviantony

This comment has been minimized.

Copy link

deviantony commented Feb 25, 2015

@driskell 's alternative is great but it also lack the presence of a debian repository, which is mainly why I didn't switch to it.

@pnegahdar

This comment has been minimized.

Copy link

pnegahdar commented Feb 25, 2015

@Siljanovski yup, pretty much what I'm doing. Careful with the the wget on master you should use "https://github.com/elasticsearch/logstash-forwarder/archive/v0.3.1.tar.gz" instead. Also the touch should be unnecessary with the echo > creates if DNE.

@amosshapira

This comment has been minimized.

Copy link

amosshapira commented Feb 25, 2015

What I ended up doing for now is to salvage a copy of the .deb file from the cache of one of our running servers and build a repository around it on S3 using reprepro. Eventually I'll get around to mirror all such "small" external repositories we use, and would recommend everyone else to do the same.

If you need a copy of the repository then let me know how you want me to get it to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.