Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(docker source): Enrich events with metadata #1149

Merged
merged 18 commits into from
Nov 26, 2019
Merged

enhancement(docker source): Enrich events with metadata #1149

merged 18 commits into from
Nov 26, 2019

Conversation

ktff
Copy link
Contributor

@ktff ktff commented Nov 5, 2019

Closes #1042

There is one open question:

Currently, metadata is coming from two api endpoints, container list and container inspect. They are returning slightly different data:

  • created_at, container list returns UNIX timestamp, container inspect returns rfc3339 timestamp
  • names, container list returns list of names, container inspect returns one name

This will result in containers having different metadata depending on when was vector started relative to it.

This could be avoided if we only use metadata from container inspect endpoint. With the cost of sometimes additional call to docker api, and loss of extra names. As a side effect, event metadata would instead of list of names have one name.

Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
src/sources/docker.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@LucioFranco LucioFranco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see the task unififcation change but otherwise this looks great.

src/sources/docker.rs Outdated Show resolved Hide resolved
src/sources/docker.rs Outdated Show resolved Hide resolved
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
@ktff
Copy link
Contributor Author

ktff commented Nov 6, 2019

@LucioFranco and what do you think about the above question regarding different metadata?

  - Delete containers after logs have been collected
 - Avoid creating same named containers

Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
@LucioFranco
Copy link
Contributor

@ktff oops, I thought I answered that! I say go with the simple solution of calling inspect when we need it. We should be able to cache the result and it shouldn't be expensive at all.

Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
@ktff
Copy link
Contributor Author

ktff commented Nov 7, 2019

With the last commit, events will be enriched with one container name, instead of a list of container names.

Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
Signed-off-by: Kruno Tomola Fabro <krunotf@gmail.com>
@binarylogic
Copy link
Contributor

@ktff this looks great! @LucioFranco will be back from vaction next week and we can get this reviewed one last time.

@binarylogic
Copy link
Contributor

Noting, just assigning @LucioFranco for review which will happen next week.

Copy link
Contributor

@LucioFranco LucioFranco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@binarylogic binarylogic self-requested a review November 25, 2019 17:38
Copy link
Contributor

@binarylogic binarylogic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ktff this looks great! I'd like to make some changes to the metadata fields.

  • The docker specific metadata fields should not reside in event/mod.rs since they are specific to the docker source. Can we remove them from there? Specifically, the STREAM, IMAGE, CONTAINER,CREATED_AT and NAME fields.
  • I find some of the naming ambiguous. Can we rename the fields to: container_id, container_name, container_created_at, image, andstream.

I am updating the docs now to reflect these changes. Once done we should be good to merge.

Signed-off-by: Ben Johnson <bjohnson@binarylogic.com>
@binarylogic
Copy link
Contributor

I've updated the documentation to reflect the above changes. Once they're made in the code this should be good to merge.

binarylogic and others added 4 commits November 25, 2019 13:22
Signed-off-by: Ben Johnson <bjohnson@binarylogic.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
@LucioFranco LucioFranco merged commit f20fc4a into vectordotdev:master Nov 26, 2019
@ktff ktff deleted the docker_metadata branch November 26, 2019 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve docker source's event metadata
3 participants