Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
METRON-939: Upgrade ElasticSearch and Kibana #840
EDIT This is ready now (11/30/17)
This PR builds on the great work in #619
I wanted to get eyes on this despite some outstanding work left. It was no small undertaking moving 3 major versions, so there's quite a bit to review. In the future I would like to recommend we make feature branches for changes of this magnitude.
This work upgrades Elasticsearch and Kibana both to 5.6. I'm running a local full dev build now, but assuming no failures this will spin up full dev with everything, including the Kibana dashboard (had to be completely rewritten).
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:
For code changes:
For documentation related changes:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
I don't think that's a bad idea. I definitely like having test scripts associated with the PR's because we can explicitly see what was done and/or recommended at the time the PR was being ushered into master. But we could also start to consolidate these into manual testing scripts that we reference ongoing. Then you could say something like:
That could be links, or we could copy-paste the current manual test plan from the source tree. I think the main thing I would want is to have the PR's test plan (here in the comments) be maintained statically for posterity based on the code at that time. For example, since we're removing the data pruner, the new test plan would not have a data pruner test, but I'd want to make sure any earlier PRs that did have a test plan for it can still be viewed statically. Basically, you'd either include the latest commit as part of the link or copy-paste the test plan.
Yeah, I'm trying to think of the right way to do it, without having to have it be "officially maintained".
Something like that. Then people could edit them and adapt them etc.
Just a status update on this. We're currently waiting for 0.4.2 to roll out before this gets committed. We definitely want more eyes and testing on this PR considering its breadth and size. We do not have any +1's yet, and I would prefer to have at least 2 for good measure, if at all possible.
I spun this up in full dev and spent all day testing it. From a functional perspective, I can not find anything wrong with it. I ran through the test plan in this PR and everything worked as expected. I also tested this exhaustively with the Alerts UI and Swagger UI and everything works great.
The only issue I found are with the Alerts UI e2e tests. These no longer run at all and I suspect it's because the templates have changed with ES 5.6.2 so loading e2e test data and template no longer works. I'm not sure that this should hold up this PR since e2e tests are actively being worked on in #857 but I wanted everyone to be aware.
I ran this up in full dev again and verified the e2e tests now work similar to how they do in master. I also manually tested several other areas including the Alerts UI, Kibana and Swagger. Everything works as expected. Assuming others are still testing but +1 from me.