hubble observe fails with "requested data has been overwritten and is no longer available" #17036
Labels
Comments
michi-covalent
added
kind/bug
michi-covalent
added a commit
to michi-covalent/cilium
that referenced
this issue
Aug 4, 2021
Currently GetFlows() fails with the following error when a position in
the ring buffer being read by Ring.read() has been overwritten:
requested data has been overwritten and is no longer available
This turned out to be impractical as it makes it difficult to read all
the flows in the ring buffer (e.g.. hubble observe --all). GetFlows()
would fail if Hubble observes a single flow between the reader rewinding
to the oldest position and retrieving the entry.
This patch modifies Ring.read() so that GetFlows() returns LostEvent
instead of stopping with an error. The caller of GetFlows() can then
decide how to handle LostEvent.
Note that this makes the behavior of Ring.read() consistent with that
of Ring.readFrom() used in the follow mode. It generates LostEvent and
continues following instead of failing with ErrInvalidRead.
Fixes: cilium#17036
Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
ti-mo
pushed a commit
that referenced
this issue
Aug 18, 2021
Currently GetFlows() fails with the following error when a position in
the ring buffer being read by Ring.read() has been overwritten:
requested data has been overwritten and is no longer available
This turned out to be impractical as it makes it difficult to read all
the flows in the ring buffer (e.g.. hubble observe --all). GetFlows()
would fail if Hubble observes a single flow between the reader rewinding
to the oldest position and retrieving the entry.
This patch modifies Ring.read() so that GetFlows() returns LostEvent
instead of stopping with an error. The caller of GetFlows() can then
decide how to handle LostEvent.
Note that this makes the behavior of Ring.read() consistent with that
of Ring.readFrom() used in the follow mode. It generates LostEvent and
continues following instead of failing with ErrInvalidRead.
Fixes: #17036
Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
ti-mo
pushed a commit
to ti-mo/cilium
that referenced
this issue
Aug 18, 2021
[ upstream commit 98b5fb3 ] Currently GetFlows() fails with the following error when a position in the ring buffer being read by Ring.read() has been overwritten: requested data has been overwritten and is no longer available This turned out to be impractical as it makes it difficult to read all the flows in the ring buffer (e.g.. hubble observe --all). GetFlows() would fail if Hubble observes a single flow between the reader rewinding to the oldest position and retrieving the entry. This patch modifies Ring.read() so that GetFlows() returns LostEvent instead of stopping with an error. The caller of GetFlows() can then decide how to handle LostEvent. Note that this makes the behavior of Ring.read() consistent with that of Ring.readFrom() used in the follow mode. It generates LostEvent and continues following instead of failing with ErrInvalidRead. Fixes: cilium#17036 Signed-off-by: Michi Mutsuzaki <michi@isovalent.com> Signed-off-by: Timo Beckers <timo@isovalent.com>
tklauser
pushed a commit
that referenced
this issue
Aug 19, 2021
[ upstream commit 98b5fb3 ] Currently GetFlows() fails with the following error when a position in the ring buffer being read by Ring.read() has been overwritten: requested data has been overwritten and is no longer available This turned out to be impractical as it makes it difficult to read all the flows in the ring buffer (e.g.. hubble observe --all). GetFlows() would fail if Hubble observes a single flow between the reader rewinding to the oldest position and retrieving the entry. This patch modifies Ring.read() so that GetFlows() returns LostEvent instead of stopping with an error. The caller of GetFlows() can then decide how to handle LostEvent. Note that this makes the behavior of Ring.read() consistent with that of Ring.readFrom() used in the follow mode. It generates LostEvent and continues following instead of failing with ErrInvalidRead. Fixes: #17036 Signed-off-by: Michi Mutsuzaki <michi@isovalent.com> Signed-off-by: Timo Beckers <timo@isovalent.com>
krishgobinath
pushed a commit
to krishgobinath/cilium
that referenced
this issue
Oct 20, 2021
Currently GetFlows() fails with the following error when a position in
the ring buffer being read by Ring.read() has been overwritten:
requested data has been overwritten and is no longer available
This turned out to be impractical as it makes it difficult to read all
the flows in the ring buffer (e.g.. hubble observe --all). GetFlows()
would fail if Hubble observes a single flow between the reader rewinding
to the oldest position and retrieving the entry.
This patch modifies Ring.read() so that GetFlows() returns LostEvent
instead of stopping with an error. The caller of GetFlows() can then
decide how to handle LostEvent.
Note that this makes the behavior of Ring.read() consistent with that
of Ring.readFrom() used in the follow mode. It generates LostEvent and
continues following instead of failing with ErrInvalidRead.
Fixes: cilium#17036
Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
cilium version: v1.10.3
i'm hitting this error fairly consistently on nodes with moderate traffic:
ref: there is a draft PR to re-implement the ring buffer: #14304
The text was updated successfully, but these errors were encountered: