Create copies of net.IP and net.HardwareAddr when decoding IE values #330
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When using the provided slice (provided by the collector) directly, the garbage collector will not be able to release the underlying array while references to the net objects exist. The underlying array is allocated when reading the message from the TCP connection and can be 100s of bytes in size. The collector will typically run aggregation and keep references to the data record and its IEs for some time. During that time, we cannot release the packet buffer, while new packets keep coming in. To avoid the issue, we create a new slice, with a new underlying array, and the packet buffer can be releases as soon as all IEs are instantiated. Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas
force-pushed
the
copy-net-addresses-in-DecodeAndCreateInfoElementWithValue
branch
from
c6e2cf9
to
77ae683
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #330 +/- ##
==========================================
- Coverage 73.53% 73.39% -0.14%
==========================================
Files 19 19
Lines 2800 2808 +8
==========================================
+ Hits 2059 2061 +2
- Misses 574 578 +4
- Partials 167 169 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Before the change: After the change: The difference is roughly 33KB (1,000 packets x 33B per packet). |
heanlan
reviewed
Oct 4, 2023
|
|
||
| const numRecords = 1000 | ||
|
|
||
| ies := make([]entities.InfoElementWithValue, 0, numRecords) |
There was a problem hiding this comment.
Apology if I didn't fully understand the test. Is there a need to collect and validate the sourceIPv4Address ie values?
There was a problem hiding this comment.
I'll write a comment. If these are not collected / used after calling runtime.ReadMemStats, everything will be GC'd.
Signed-off-by: Antonin Bas <abas@vmware.com>
heanlan
pushed a commit
to heanlan/go-ipfix
that referenced
this pull request
Dec 7, 2023
…mware#330) When using the provided slice (provided by the collector) directly, the garbage collector will not be able to release the underlying array while references to the net objects exist. The underlying array is allocated when reading the message from the TCP connection and can be 100s of bytes in size. The collector will typically run aggregation and keep references to the data record and its IEs for some time. During that time, we cannot release the packet buffer, while new packets keep coming in. To avoid the issue, we create a new slice, with a new underlying array, and the packet buffer can be releases as soon as all IEs are instantiated. Signed-off-by: Antonin Bas <abas@vmware.com>
heanlan
pushed a commit
that referenced
this pull request
Dec 7, 2023
…330) When using the provided slice (provided by the collector) directly, the garbage collector will not be able to release the underlying array while references to the net objects exist. The underlying array is allocated when reading the message from the TCP connection and can be 100s of bytes in size. The collector will typically run aggregation and keep references to the data record and its IEs for some time. During that time, we cannot release the packet buffer, while new packets keep coming in. To avoid the issue, we create a new slice, with a new underlying array, and the packet buffer can be releases as soon as all IEs are instantiated. Signed-off-by: Antonin Bas <abas@vmware.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using the provided slice (provided by the collector) directly, the garbage collector will not be able to release the underlying array while references to the net objects exist. The underlying array is allocated when reading the message from the TCP connection and can be 100s of bytes in size. The collector will typically run aggregation and keep references to the data record and its IEs for some time. During that time, we cannot release the packet buffer, while new packets keep coming in. To avoid the issue, we create a new slice, with a new underlying array, and the packet buffer can be releases as soon as all IEs are instantiated.