Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading flows from stdin #524

Merged
merged 1 commit into from
Apr 7, 2021
Merged

Support reading flows from stdin #524

merged 1 commit into from
Apr 7, 2021

Conversation

michi-covalent
Copy link
Collaborator

@michi-covalent michi-covalent commented Apr 2, 2021

Implement an ObserverClient that reads jsonpb flows from an io.Reader
instead of connecting to Hubble, and modify hubble observe command
to use it when stdin is a pipe. For example,

hubble observe -o jsonpb --last 1000 > flows.log
cat flows.log | hubble observe -n kube-system

A few things to note for observing flows from stdin:

  • --follow, --last, and server flags are ignored.
  • Flows are processed in the order they are read from stdin without
    sorting them by timestamp.

Fixes #523

Signed-off-by: Michi Mutsuzaki michi@isovalent.com

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label PR is blocked until the release note is set label Apr 2, 2021
@michi-covalent michi-covalent added release-note/minor This PR introduces functionality that users may find relevant to operating Hubble. ⌨️ area/cli Impacts the command line interface of any command in the repository. labels Apr 2, 2021
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label PR is blocked until the release note is set label Apr 2, 2021
Copy link
Member

@rolinh rolinh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty neat, I like it! 🚀 Please, see some suggestions for improvements below.

Have you considered reading from a (several) file(s) as well? Maybe via optional positional arguments to the observe command.

cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
Copy link
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! 🚀 Just one question in addition to what Robin already pointed out.

cmd/observe/io_reader_observer.go Show resolved Hide resolved
@michi-covalent
Copy link
Collaborator Author

Have you considered reading from a (several) file(s) as well? Maybe via optional positional arguments to the observe command.

reading from multiple files are already supported in a way. you can do:

cat *.log | hubble observe

you can also pre-process files before sending to hubble observe. for example if flow.log files are compressed:

zcat *.log | hubble observe

we could add positional arguments, but i thought it might get a bit more confusing to users. right now the separation is pretty clear i think: if stdin is a pipe, read flows from there. else connect to hubble server.

@michi-covalent michi-covalent force-pushed the pr/michi/pipe branch 2 times, most recently from b24842e to 28754b2 Compare April 7, 2021 03:43
Copy link
Member

@rolinh rolinh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm besides a missing error check.

cmd/observe/io_reader_observer.go Show resolved Hide resolved
Implement an ObserverClient that reads jsonpb flows from an io.Reader
instead of connecting to Hubble, and modify `hubble observe` command
to use it when stdin is a pipe. For example,

    hubble observe -o jsonpb --last 1000 > flows.log
    cat flows.log | hubble observe -n kube-system

A few things to note for observing flows from stdin:

- --follow, --last, and server flags are ignored.
- Flows are processed in the order they are read from stdin without
  sorting them by timestamp.

Fixes #523

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
@michi-covalent michi-covalent merged commit d419ce1 into master Apr 7, 2021
@michi-covalent michi-covalent deleted the pr/michi/pipe branch April 7, 2021 17:35
pchaigno added a commit to pchaigno/cilium that referenced this pull request May 7, 2021
hubble observe now supports [1] reading flows from stdin if they are in
jsonpb format. We should therefore emit jsonpb artifact files for tests,
to be able to feed those files back into hubble observe. That will allow
us to use hubble observe's filters and different output format.

1 - cilium/hubble#524
Signed-off-by: Paul Chaignon <paul@cilium.io>
ti-mo pushed a commit to cilium/cilium that referenced this pull request May 10, 2021
hubble observe now supports [1] reading flows from stdin if they are in
jsonpb format. We should therefore emit jsonpb artifact files for tests,
to be able to feed those files back into hubble observe. That will allow
us to use hubble observe's filters and different output format.

1 - cilium/hubble#524
Signed-off-by: Paul Chaignon <paul@cilium.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⌨️ area/cli Impacts the command line interface of any command in the repository. release-note/minor This PR introduces functionality that users may find relevant to operating Hubble.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

hubble observe: support reading from json file
6 participants