-
Notifications
You must be signed in to change notification settings - Fork 191
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Containerization of PFELK #69
Comments
I agree with the docket route. This is something I’ve been planning on
pursuing but haven’t had the time. This weekend I have about a day’s worth
of time, I plan to work on this.
…On Thu, Jan 16, 2020 at 06:33 fktkrt ***@***.***> wrote:
*Is your feature request related to a problem? Please describe.*
Running pfelk in containers could be another deploy method.
*Describe the solution you'd like*
There would be Dockerfiles for the components (Elasticsearch, Logstash,
Kibana), the configuration files, patterns would be included in them.
Management is question to discuss, we could use simple docker compose for
configuring multiple containers, or we can choose an orchestration tool eg.
Kubernetes or Docker Swarm.
Should we target a single or multi node architecture?
In my opinion we should either stick to Docker Compose or choose
Kubernetes, depending on the architecture.
*Additional context*
I am in favor of using VMs under the ELK stack, but being able to deploy
this on containers could be better suited for some use cases.
I am quite busy at the moment, but would like to work in this, so any help
is appreciated starting from design decisions to implementation details.
What everybody is thinking?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/a3ilson/pfelk/issues/69?email_source=notifications&email_token=AEA2HR54LIQVIM4PN7DZQDTQ6BAYNA5CNFSM4KHSP6U2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IGTYYMA>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEA2HR64DLL3VNXMMBMJOUTQ6BAYNANCNFSM4KHSP6UQ>
.
|
I'm happy to hear that. Don't hesitate to tell me if I can assist you in any way. |
Leaning towards LXC after initial attempt. Dockers are single processes only (one process for Elastic, Logstash, and Kibana). However, incorporating cron jobs (MaxMind) and such are problematic whereas LXC are easier alternative for this specific endeavor. |
In my opinion LXC could scare people away, for most people containers are docker, full stop. We can use docker compose to pull up individual containers to each services (elasticsearch, logstash, kibana). I think that would be the official recommendation for this type of solution. |
I'm thinking fine-tuning something like this: https://github.com/elastic/stack-docker |
My initial thoughts with the docker is the lack of umph (resources). It
drags (very sluggish) and unlikely an advantageous solution for
implementation. I'll keep tinkering but not overly comfortable
implementing it.
…On Sun, Jan 19, 2020 at 12:31 PM fktkrt ***@***.***> wrote:
I'm thinking something like this: https://github.com/elastic/stack-docker
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/a3ilson/pfelk/issues/69?email_source=notifications&email_token=AEA2HR2KDABXZPEGZW3XOIDQ6SE5PA5CNFSM4KHSP6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKXTQA#issuecomment-576027072>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEA2HR2ZS5EXHT64EYIV6WLQ6SE5PANCNFSM4KHSP6UQ>
.
|
Should we test/measure different approaches then maybe? |
I’d also suggest it be Docker or bust. If deploying to Kubernetes, consider leveraging this Elastic operator to handle the ELK stack deployment. You’d just need to insert the Pf specific configurations. Those can generally be defined in code as well as part of the values that get injected. |
Cron job ideas:
|
Currently testing with LXC; and working (tweaking) the Docker instance. |
I can manage building Docker samples for testing to take some workload of you, just let me know. |
Perfect! I only have time during the weekend, at the moment. I pulled the
latest dockers from elastic... rebuilt Logstash and still tweaking but feel
free to create/build.
…On Wed, Jan 22, 2020 at 10:30 fktkrt ***@***.***> wrote:
I can manage building Docker samples for testing to take some workload of
you, just let me know.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/a3ilson/pfelk/issues/69?email_source=notifications&email_token=AEA2HR7Y4SEU4DWXIS5ZERTQ7BRB5A5CNFSM4KHSP6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJUAECQ#issuecomment-577241610>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEA2HR64RRCNVOEPXBLKHYDQ7BRB5ANCNFSM4KHSP6UQ>
.
|
My only experience with docker was through the modules in openmediavault - but once you have something ready to test, i'd be glad to mess around with it. |
@revere521 - thanks! Pending weekend plans, I hope to have something operational this weekend. |
I have a new branch with an initial attempt using Docker, you can check it here: https://github.com/fktkrt/pfelk/archive/docker-pfelk.zip Some info for faster testing:
Currently it is bringing up the three services, populating the logstash filters and binding the services together. At the moment, the Other debug options:
If everything is set up, you should see a similar output:
Check the elasticsearch nodes:
|
i think i'm going to look at setting up docker on server essentials 2016 so i can help test. I have a VM that i'll try these with first; then i can possibly try it on my physical server. Maybe this weekend |
It looks like essentials may not have the necessary components for docker based on a cursory search...so i built an ubuntu 18.04.3 server VM and setup docker there. ready to test, but not tonight :) |
Just finished setting up automated building & testing with Travis-CI, you can check it out at: https://travis-ci.org/fktkrt/pfelk or in the README at https://github.com/fktkrt/pfelk I think we should incorporate this in the original repo, if we go this way. What do you think? |
the Travis-CI looks pretty neat for sure...that tests PRs on the fly by running your stuff out in the aether somewhere and giving you a pass/fail? To test this .. lets pretend i'm just some old guy on the internet (cough,cough)... It looks like:
Seriously though - for step 2 - do i only need to clone whats int the docker-pfelk folder in the repo on my docker machine, then run |
Yes, in a nutshell you only need a
Yes, if you have docker engine and compose installed, that should get you started. |
i got this when on my first attempt:
i can see in the .yml where it calls the variable, but not sure where to set it? |
It should be set under |
yes, it was probably my error - i re-cloned the repo, then an the docker-compose command with sudo and its installing now. |
@revere521, was is successful? |
Sorry, its beena busy couple of days - i justy got back to chekc and this is my output:
I will try again this evening |
Sorry, I let the installs run and didnt get back to it last night. I'll check later this evening when I back in front of my machine
On Jan 27, 2020 10:11 AM, fktkrt <notifications@github.com> wrote:
@revere521<https://github.com/revere521>, was is successful?
@a3ilson<https://github.com/a3ilson>, have you got any success with lxc or either with your docker or with mine? Any thoughts on integrating Travis-CI?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/a3ilson/pfelk/issues/69?email_source=notifications&email_token=AFTYA6P7FA3UGFDP3EHWHGTQ732QPA5CNFSM4KHSP6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ73E4Q#issuecomment-578794098>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFTYA6LXSR44DBXWEEHSYGTQ732QPANCNFSM4KHSP6UQ>.
|
Ok, it installed successfully the 2nd time - i can hit the default Kibana web interface. Just figured out that i needed to edit the configs under the logstash/pipline folder. Still figuring out how to send data to it |
You only need to configure the IP address of your docker host with port 5140 as your pfSense remote host, at "Enable Remote Logging". |
I did add it as a second remote log server in pfsense, but it doesn't seem to be ingesting data yet...i'll need to troubleshoot if thats a network issue |
for some reason port 5140 is not open, or at least not reachable. I made sure it wasn't a UFW firewall issue - and that doesn't seem to be the case. from the port test in pfsense i can hit all the ports you have open for Elasticsearch and Kibana (in the docker-compose.yml); and i can connect to port 9600 for Logstash, but not port 5000 - and i added 5140 and cant hit that either. |
@revere521 - I would recommend doing a tcpdump on port 5140. This will help troubleshoot (i.e. are the logs being sent but not parsed or not sent) |
@revere521 - I honestly haven't had any time but it's on my list of things to do. I'll finish tinkering with docker before finalizing a container. Feel free to add another install option (linking Travis CI) to the README. I haven't had a chance to evaluate your docker(s) but did you specify the pfSense/OPNsense IP address, allow for a user input to configure or omit? |
I think @fktkrt was asking - but i did have to edit the config files in the pfelk/docker/logstash/pipeline folder, then run the docker-compose commands - it looks like the config files are then copied to the correct place during the build. |
the server where i'm running docker looks like its getting the syslog data; its just not making it into the logstash container (interestingly, you can also see here my temp/humidity/light sensor spamming the thingspeak api for some reason)
|
@revere521 - Access the Docker and check the 01-inputs.conf. Specifically, line 9 which provides the host ip ( if [host] =~ /172.22.33.1/ { ). If this line is present, adjust to match your pf/OPNsense instance. Alternatively, you can omit lines 9-13 which will allow any traffic received on port 5140 to be parsed vs traffic received from a particular IP address. |
I did set the correct IP in the config file in the ./pfelk/docker/logstash/pipeline/01-inputs.conf I tried a rebuild commenting that section out, and it doesn't change the behavior. This looks like an issue with traffic on port 5140 getting from the physical network to thevirtual docker network that these three containers are configured on (in the docker-compose.yml it looks like its a bridged network called "elk") - i just don't really know how that works |
Gotcha, If I have time this weekend I’ll try to walk you through it via
YouTube tutorial.
…On Tue, Jan 28, 2020 at 20:11 revere521 ***@***.***> wrote:
I did set the correct IP in the config file in the
./pfelk/docker/logstash/pipeline/01-inputs.conf
I tried a rebuild commenting that section out, and it doesn't change the
behavior.
This looks like an issue with traffic on port 5140 getting from the
physical network to thevirtual docker network that these three containers
are configured on (in the docker-compose.yml it looks like its a bridged
network called "elk") - i just don't really know how that works
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/a3ilson/pfelk/issues/69?email_source=notifications&email_token=AEA2HR4CD4J7Z27JPUD3VVTRADJTBA5CNFSM4KHSP6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKFTD5Y#issuecomment-579547639>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEA2HR35Y6OJRU4MUOU2LDTRADJTBANCNFSM4KHSP6UQ>
.
|
Travis-CI: @a3ilson, I was thinking more like incorporating this as a fail/pass badge testing. Should I include my repository's Travis branch for this? Should I PR my docker changes for review, then tinker that later? We can even add multiple deploy methods for docker and lxc. @revere521, yes you are right, port 5140/udp was not exposed in the logstash container, it's fixed now. the CI build is passing, will test properly later this evening. |
For some reason, even after making that edit in the docker-compose.yml and running
the documentation for docker compose states that the build command is build and rebuild - is that the right thing to do? |
ok, i was finally able to get it to pickup the port change by running: I also added
I'm getting data now, and i'll let it collect data over the course of today and see whats-what later |
I got a working and tested docker with a makeshift GeoIP configured, for now. All required files are posted and will author initial configuration/installation instructions within the week. |
Looks fine to me!
|
Will do - I'll spend more time on it next weekend. |
@fktkrt feel free to test with Travis-CI |
Travis-CI now supported at org level. Added the pass/failure badge to the README as well. |
This was closed and opened within the Docker-pfelk under issue #2. The only outlier is to accomplish the cron job for GeoIP within the docker. |
Is your feature request related to a problem? Please describe.
Running pfelk in containers could be another deploy method.
Describe the solution you'd like
There would be Dockerfiles for the components (Elasticsearch, Logstash, Kibana), the configuration files, patterns would be included in them.
Management is question to discuss, we could use simple docker compose for configuring multiple containers, or we can choose an orchestration tool eg. Kubernetes or Docker Swarm.
Should we target a single or multi node architecture?
In my opinion we should either stick to Docker Compose or choose Kubernetes, depending on the architecture.
Additional context
I am in favor of using VMs under the ELK stack, but being able to deploy this on containers could be better suited for some use cases.
I am quite busy at the moment, but would like to work in this, so any help is appreciated starting from design decisions to implementation details.
What everybody is thinking?
The text was updated successfully, but these errors were encountered: