New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docker Integration #632
Add docker Integration #632
Conversation
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪 |
The CI issue is I think unrelated? |
Pinging @elastic/integrations (Team:Integrations) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments.
@fearful-symmetry how about adding system tests for this one? It will help with catching cases of missing fields etc. I'm not sure if this can happen with connecting to an actual docker socket ( @mtojek wdyt?) but in any case a mock API could be used so as to mock the response and being able to validate the package. There is a similar case in k8s package from which you can reuse the mock json api.
I wonder if we can permanently mount the
If we can locate the sock path in every OS (Windows/Mac/Linux), then we can permanently set it there. |
So, I know the path is the same on Darwin. Not sure about windows. Not sure if we'd want to? On Linux at least, we also need to run with Also, do we have a guide somewhere for system tests and using whatever mock apis we have? |
Yah @ChrsMark could you explain the mock stuff for k8s? |
Okay, think I found the system test docs. Is there a reason they're all in |
CI failures look unrelated again. So, Not sure how testing for this would work. Since this runs in CI, can our CI environment even access a docker daemon socket? |
PR adding system tests for k8s using the mock APIs: #569 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you rebase this branch against the master branch, the CI will build only changes in this package (diff).
Answering your question - why is the documentation in elastic-package
. Test runners are part of this tool, so they're included in tool's documentation. Besides this we have also updated the CONTRIBUTING guide for Integrations repo (incl. table of contents). I believe they should be easier to find.
Regarding the --priviledged
and --user=root
option, it would be great if you can perform short research on the CI side (experimental PR?).
3718127
to
16d3dd5
Compare
Alright, I'm gonna take a look at the moked K8s tests and see what I can figure out. |
Alright, so it looks like the way to do this is to actually spin up a server that can mock whatever the API is serving up. In which case, I'd need to mock the unix socket or TCP endpoint. |
So, experimenting more with mocking the docker tests, and I want some opinions here. I'd need something semi-interactive, since the docker client will try to grab data on different containers, and hit a few APIs. It looks like the k8s tests rely on another container to mock a Prometheus endpoint. I'd need to do something similar, and write some custom python or a custom container to actually mock the API behavior itself. @ChrsMark any opinions here? |
@fearful-symmetry yeap, I think what you explain should be ok. In my case with k8s there were only static endpoints that I had to mock, return Prometheus and json responses. I think that if you are able to completely mock the full set of endpoints that docker client is gonna hit you should be fine. There are plenty of ready to use json mock servers out there so maybe you avoid writing your own python server. Also have a look into the nginx reverse proxy (in docker-compose.yml) which can actually redirect any specific request to any backend endpoint giving you some more flexibility. However, I'm not sure how easy it would be if requests are not deterministic. Happy to chat about it more if needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we release as beta
instead of experimental
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add unit
s and metric_type
s to fields definitions when possible (as in #402).
@ChrsMark / @mtojek can someone help me out with the system tests? I have a mock system setup, but I can't seem to run a test locally. Running the test command in the
The |
Would you mind pushing up these files to the PR so we can pull down the PR and try to repro what you're seeing with the system tests locally? Thanks! |
Try this - jump out of the |
@ycombinator / @mtojek pushed a single system test here. We might want to improve the docs for these, since a lot of it is fairly abstract and I had to guess what should go in the docker-compose file. |
25b7807
to
b74a92e
Compare
@mtojek So, the only doc bug I've found is the issue with naming |
I see, it seems like the documentation is bit outdated in this area. I will update it today. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (only one nit-pick)
@jsoriano still getting test failures with de-dot off and the mapping changed from |
Yes, if we continue using |
Okay, now I'm running into issues with de-dotting and the object mapping:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* add docker Integration * update ecs, add fields and screenshot * update fields, release as beta * run format * try out systems tests * run format * setup system tests * version as a beta release * disable de-dot * update object types, add de-dot * add changelog * format
What does this PR do?
This adds the Docker integration to Fleet, with all the metrics from the metricbeat docker module.
Checklist
How to test this PR locally
elastic-package
, you'll need to modify the docker-compose to mount the docker socket and run as root:Screenshots