This repository was archived by the owner on Nov 24, 2025. It is now read-only.
Adds support for CSV parsing in astats#4993
Merged
mattjackson220 merged 4 commits intoapache:masterfrom Sep 2, 2020
ezelkow1:tm-csv
Merged
Adds support for CSV parsing in astats#4993mattjackson220 merged 4 commits intoapache:masterfrom ezelkow1:tm-csv
mattjackson220 merged 4 commits intoapache:masterfrom
ezelkow1:tm-csv
Conversation
The astats parser will check the incoming content-type header to determine if it should parse the data as json or csv. Added a cfg option of http_polling_format that defaults to "text/json", this is what is sent on http requests to caches. Changing it to "text/csv" will enable the csv output if the cache's astats plugin as has support for CSV, otherwise astats will respond with "text/json" or "text/javascript" depending on the version.
zrhoffman
suggested changes
Aug 31, 2020
zrhoffman
reviewed
Aug 31, 2020
…just care that parsing succeeded
Now scans in all data in to a string slice which then lets us allocate the data map up front. In benches this bumps the performance up from ~8000 to 10000
Member
Author
|
Update to use new allocation method to do the data map allocation all up front, turns out this is more performant reading all of the string data in at once to one dynamically allocated block and then basing the size of the map on that so that the map does not have to be continually resized BenchmarkAstatsJson-8 5836 197696 ns/op 134473 B/op 2028 allocs/op |
zrhoffman
reviewed
Aug 31, 2020
zrhoffman
approved these changes
Sep 2, 2020
Member
Since this is a new feature, would you please add a changelog entry? |
Member
Author
@zrhoffman Opened #5009 |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR (Pull Request) do?
Adds CSV parsing support to traffic monitor for the astats plugin.
The astats parser will check the incoming content-type header to determine if it should parse the data as json or csv. Added a cfg option of http_polling_format that defaults to "text/json", this is what is sent on http requests to caches. Changing it to "text/csv" will enable the csv output if the cache's astats plugin as has support for CSV, otherwise astats will respond with "text/json" or "text/javascript" depending on the version.
Which Traffic Control components are affected by this PR?
What is the best way to verify this PR?
Set the http_polling_format configuration option in TM to "text/csv" to send that in the accept header to caches. Using a cache that has a recent version of astats with CSV support then verify that the cache and its stats are reported correctly in TM.
Tests exist in the traffic_monitor/cache directory
If this is a bug fix, what versions of Traffic Control are affected?
The following criteria are ALL met by this PR
Additional Information
I have also included benchmarks for comparison to json usage:
BenchmarkAstatsJson-8 5931 203527 ns/op 134475 B/op 2028 allocs/op
BenchmarkAstatsCSV-8 8680 147683 ns/op 119839 B/op 812 allocs/op
The CSV is ~40-50% faster depending on run while also greatly decreasing the amount of allocations per operation which should hopefully help with garbage collection pressures.
Updating with new benchmark numbers after changing to use upfront allocation on the data map:
BenchmarkAstatsJson-8 5836 197696 ns/op 134473 B/op 2028 allocs/op
BenchmarkAstatsCSV-8 10000 110413 ns/op 110065 B/op 800 allocs/op
The csv is now more efficient both for operations and allocations compared to json doing a single up front allocation of the map based on the number of lines returned from astats