Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hubble/parser/threefour: decode layers only if there is a packet #14448

Merged
merged 1 commit into from
Dec 21, 2020

Conversation

tklauser
Copy link
Member

When bumping github.com/google/gopacket to v1.1.19 (done as part of a
follow-up PR bumping all of Cilium's vendored dependencies),
gopacket.DecodingLayerParser.DecodeLayers will return an error such as
"Ethernet packet too small" in case the provided packet is empty. In
previous gopacket versions, a nil error was returned in this case. This
was reported upstream as google/gopacket#846.

This behavior will break tests that have no additional packet data e.g.
TestLocalObserverServer_GetFlows which uses raw monitor.TraceNotifyV0
messages.

This can e.g. be reproduced using:

$ go get github.com/google/gopacket@v1.1.19
$ go mod tidy && go mod vendor
$ go test -v -run "^TestLocalObserverServer_GetFlows$" ./pkg/hubble/observer
=== RUN   TestLocalObserverServer_GetFlows
    local_observer_test.go:131:
        	Error Trace:	local_observer_test.go:131
        	Error:      	Not equal:
    	            	expected: 0xa
    	            	actual  : 0x0
        	Test:       	TestLocalObserverServer_GetFlows
--- FAIL: TestLocalObserverServer_GetFlows (0.00s)
FAIL
exit status 1
FAIL	github.com/cilium/cilium/pkg/hubble/observer	0.066s

Avoid breaking these tests and also avoid an unnecessary call in case
there is no packet data. Once google/gopacket#846 is fixed upstream, the
check could be removed again when bumping the dependency.

@tklauser tklauser added release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay labels Dec 18, 2020
@tklauser tklauser requested review from a team and glibsm December 18, 2020 11:10
@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.10.0 Dec 18, 2020
@tklauser
Copy link
Member Author

test-me-please

Copy link
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

pkg/hubble/parser/threefour/parser.go Outdated Show resolved Hide resolved
@tklauser
Copy link
Member Author

test-me-please

@tklauser
Copy link
Member Author

tklauser commented Dec 18, 2020

retest-runtime

vm provisioning failed: https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/3088

@tklauser
Copy link
Member Author

tklauser commented Dec 18, 2020

@tklauser
Copy link
Member Author

tklauser commented Dec 18, 2020

retest-gke

failed to scale cluster

@tklauser
Copy link
Member Author

retest-gke

// Since v1.1.18, DecodeLayers returns a non-nil error for an empty packet, see
// https://github.com/google/gopacket/issues/846
// TODO: reconsider this check if the issue is fixed upstream
if len(data) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already check for data len on line 147, maybe we should combine somehow? Aside from breaking tests (are they even properly set up?) what does it mean to not have any layers in a packet?

Copy link
Member Author

@tklauser tklauser Dec 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it looks like we want to error out at line 147 because the packet is too short to contain a monitor header (for lack of a better term), but the len(data) == 0 check applies to the data after the monitor header, i.e. there is some info in the monitor header which we potentially want to report. I agree that re-slicing data makes that logic harder to follow, so checking len(data[packetOffset:]) == 0 is probably better here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re. breaking tests: To be honest, I don't know whether this can happen aside from tests, i.e. with real monitor traffic. I only noticed when bumping gopacket that said tests are failing. To me it looks like the tests could also be extended to contain some packet data. But then again, given that gopacket accepted empty packets just fine in previous versions (and will do so again if the proposed upstream fix is applied), I figured we rather follow that behavior.

pkg/hubble/parser/threefour/parser.go Outdated Show resolved Hide resolved
@tklauser tklauser force-pushed the pr/tklauser/hubble-parser-empty-packet branch from 84ba6f5 to 46280ac Compare December 18, 2020 17:14
@tklauser
Copy link
Member Author

test-me-please

Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Could you please on the way improve the comment of Decode please:

--- a/pkg/hubble/parser/threefour/parser.go
+++ b/pkg/hubble/parser/threefour/parser.go
@@ -96,7 +96,7 @@ func New(
        }, nil
 }

-// Decode decodes the data from 'payload' into 'decoded'
+// Decode decodes the data from 'data' into 'decoded'
 func (p *Parser) Decode(data []byte, decoded *pb.Flow) error {
        if data == nil || len(data) == 0 {
                return errors.ErrEmptyData

When bumping github.com/google/gopacket to v1.1.19 (done as part of a
follow-up PR bumping all of Cilium's vendored dependencies),
gopacket.DecodingLayerParser.DecodeLayers will return an error such as
"Ethernet packet too small" in case the provided packet is empty. In
previous gopacket versions, a nil error was returned in this case. This
was reported upstream as google/gopacket#846.

This behavior will break tests that have no additional packet data e.g.
TestLocalObserverServer_GetFlows which uses raw monitor.TraceNotifyV0
messages.

This can e.g. be reproduced using:

    $ go get github.com/google/gopacket@v1.1.19
    $ go mod tidy && go mod vendor
    $ go test -v -run "^TestLocalObserverServer_GetFlows$" ./pkg/hubble/observer
    === RUN   TestLocalObserverServer_GetFlows
        local_observer_test.go:131:
            	Error Trace:	local_observer_test.go:131
            	Error:      	Not equal:
        	            	expected: 0xa
        	            	actual  : 0x0
            	Test:       	TestLocalObserverServer_GetFlows
    --- FAIL: TestLocalObserverServer_GetFlows (0.00s)
    FAIL
    exit status 1
    FAIL	github.com/cilium/cilium/pkg/hubble/observer	0.066s

Avoid breaking these tests and also avoid an unnecessary call in case
there is no packet data. Once google/gopacket#846 is fixed upstream, the
check could be removed again when bumping the dependency.

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
@tklauser tklauser force-pushed the pr/tklauser/hubble-parser-empty-packet branch from 46280ac to 857f581 Compare December 21, 2020 11:26
@tklauser
Copy link
Member Author

test-me-please

@tklauser
Copy link
Member Author

tklauser commented Dec 21, 2020

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Dec 21, 2020
@vadorovsky vadorovsky merged commit 9c0939a into master Dec 21, 2020
@vadorovsky vadorovsky deleted the pr/tklauser/hubble-parser-empty-packet branch December 21, 2020 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

5 participants