Query bytes fetched in query API

[christopherzli]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Leverage the existing series stats counter field which was used for metrics, and add query bytes fetched to the field and save it as part of response

Verification

curl http://localhost:10902/api/v1/query?query='COUNT(up)'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1716515107.878,"173"]}],"analysis":{},"seriesStatsCounter":{"Series":174,"Chunks":174,"Samples":2248,"Bytes":101533}}}

pantheon-range-querier-deployment-554df4d5b7-v6pcn:/$ curl 'http://localhost:10902/api/v1/query?query=COUNT%28count_over_time%28agent_build_info%5B7d%5D%29%29'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1716585935.325,"1866"]}],"analysis":{},"seriesStatsCounter":{"Series":1866,"Chunks":27814,"Samples":3161070,"Bytes":3949278}}}

pantheon-range-querier-deployment-554df4d5b7-v6pcn:/$ curl 'http://localhost:10902/api/v1/query_range?query=COUNT%28agent_build_info%29&dedup=true&partial_response=true&start=1715981469.12&end=1716586269.12&step=2419'
{"status":"success","data":{"resultType":"matrix","result":[{"metric":{},"values":[[1715981469.12,"194"],[1715983888.12,"194"],[1715986307.12,"198"],[1715988726.12,"197"],[1715991145.12,"197"],[1715993564.12,"192"],[1715995983.12,"185"],[1715998402.12,"185"],[1716000821.12,"185"],[1716003240.12,"185"],[1716005659.12,"185"],[1716008078.12,"180"],[1716010497.12,"168"],[1716012916.12,"160"],[1716015335.12,"152"],[1716017754.12,"144"],[1716020173.12,"136"],[1716022592.12,"128"],[1716025011.12,"120"],[1716027430.12,"112"],[1716029849.12,"105"],[1716032268.12,"97"],[1716034687.12,"89"],[1716037106.12,"82"],[1716039525.12,"75"],[1716041944.12,"75"],[1716044363.12,"75"],[1716046782.12,"75"],[1716049201.12,"75"],[1716051620.12,"75"],[1716054039.12,"74"],[1716056458.12,"73"],[1716058877.12,"71"],[1716061296.12,"71"],[1716063715.12,"71"],[1716066134.12,"71"],[1716068553.12,"71"],[1716070972.12,"71"],[1716073391.12,"71"],[1716075810.12,"71"],[1716078229.12,"71"],[1716080648.12,"71"],[1716083067.12,"71"],[1716085486.12,"71"],[1716087905.12,"71"],[1716090324.12,"71"],[1716092743.12,"72"],[1716095162.12,"71"],[1716097581.12,"71"],[1716100000.12,"71"],[1716102419.12,"71"],[1716104838.12,"71"],[1716107257.12,"71"],[1716109676.12,"78"],[1716112095.12,"75"],[1716114514.12,"75"],[1716116933.12,"75"],[1716119352.12,"75"],[1716121771.12,"75"],[1716124190.12,"75"],[1716126609.12,"75"],[1716129028.12,"75"],[1716131447.12,"77"],[1716133866.12,"75"],[1716136285.12,"75"],[1716138704.12,"75"],[1716141123.12,"80"],[1716143542.12,"85"],[1716145961.12,"96"],[1716148380.12,"105"],[1716150799.12,"109"],[1716153218.12,"119"],[1716155637.12,"124"],[1716158056.12,"131"],[1716160475.12,"139"],[1716162894.12,"145"],[1716165313.12,"153"],[1716167732.12,"157"],[1716170151.12,"168"],[1716172570.12,"167"],[1716174989.12,"167"],[1716177408.12,"167"],[1716179827.12,"167"],[1716182246.12,"167"],[1716184665.12,"167"],[1716187084.12,"167"],[1716189503.12,"166"],[1716191922.12,"166"],[1716194341.12,"166"],[1716196760.12,"165"],[1716199179.12,"165"],[1716201598.12,"168"],[1716204017.12,"168"],[1716206436.12,"167"],[1716208855.12,"165"],[1716211274.12,"165"],[1716213693.12,"165"],[1716216112.12,"171"],[1716218531.12,"172"],[1716220950.12,"168"],[1716223369.12,"162"],[1716225788.12,"169"],[1716228207.12,"184"],[1716230626.12,"170"],[1716233045.12,"168"],[1716235464.12,"170"],[1716237883.12,"178"],[1716240302.12,"176"],[1716242721.12,"174"],[1716245140.12,"173"],[1716247559.12,"173"],[1716249978.12,"180"],[1716252397.12,"179"],[1716254816.12,"179"],[1716257235.12,"179"],[1716259654.12,"177"],[1716262073.12,"177"],[1716264492.12,"176"],[1716266911.12,"176"],[1716269330.12,"176"],[1716271749.12,"175"],[1716274168.12,"173"],[1716276587.12,"171"],[1716279006.12,"171"],[1716281425.12,"171"],[1716283844.12,"171"],[1716286263.12,"173"],[1716288682.12,"172"],[1716291101.12,"173"],[1716293520.12,"172"],[1716295939.12,"172"],[1716298358.12,"172"],[1716300777.12,"172"],[1716303196.12,"180"],[1716305615.12,"176"],[1716308034.12,"172"],[1716310453.12,"173"],[1716312872.12,"173"],[1716315291.12,"289"],[1716317710.12,"175"],[1716320129.12,"177"],[1716322548.12,"174"],[1716324967.12,"179"],[1716327386.12,"178"],[1716329805.12,"178"],[1716332224.12,"179"],[1716334643.12,"179"],[1716337062.12,"180"],[1716339481.12,"181"],[1716341900.12,"183"],[1716344319.12,"184"],[1716346738.12,"182"],[1716349157.12,"183"],[1716351576.12,"183"],[1716353995.12,"183"],[1716356414.12,"182"],[1716358833.12,"184"],[1716361252.12,"182"],[1716363671.12,"183"],[1716366090.12,"181"],[1716368509.12,"181"],[1716370928.12,"180"],[1716373347.12,"182"],[1716375766.12,"182"],[1716378185.12,"182"],[1716380604.12,"182"],[1716383023.12,"182"],[1716385442.12,"181"],[1716387861.12,"180"],[1716390280.12,"193"],[1716392699.12,"186"],[1716395118.12,"183"],[1716397537.12,"182"],[1716399956.12,"187"],[1716402375.12,"182"],[1716404794.12,"182"],[1716407213.12,"181"],[1716409632.12,"181"],[1716412051.12,"180"],[1716414470.12,"180"],[1716416889.12,"181"],[1716419308.12,"180"],[1716421727.12,"180"],[1716424146.12,"182"],[1716426565.12,"181"],[1716428984.12,"183"],[1716431403.12,"188"],[1716433822.12,"185"],[1716436241.12,"185"],[1716438660.12,"183"],[1716441079.12,"183"],[1716443498.12,"318"],[1716445917.12,"186"],[1716448336.12,"184"],[1716450755.12,"184"],[1716453174.12,"183"],[1716455593.12,"183"],[1716458012.12,"183"],[1716460431.12,"183"],[1716462850.12,"183"],[1716465269.12,"183"],[1716467688.12,"183"],[1716470107.12,"183"],[1716472526.12,"183"],[1716474945.12,"183"],[1716477364.12,"194"],[1716479783.12,"183"],[1716482202.12,"182"],[1716484621.12,"182"],[1716487040.12,"181"],[1716489459.12,"182"],[1716491878.12,"180"],[1716494297.12,"178"],[1716496716.12,"176"],[1716499135.12,"176"],[1716501554.12,"176"],[1716503973.12,"181"],[1716506392.12,"182"],[1716508811.12,"178"],[1716511230.12,"179"],[1716513649.12,"179"],[1716516068.12,"179"],[1716518487.12,"178"],[1716520906.12,"177"],[1716523325.12,"177"],[1716525744.12,"177"],[1716528163.12,"177"],[1716530582.12,"176"],[1716533001.12,"176"],[1716535420.12,"176"],[1716537839.12,"177"],[1716540258.12,"176"],[1716542677.12,"176"],[1716545096.12,"177"],[1716547515.12,"178"],[1716549934.12,"178"],[1716552353.12,"176"],[1716554772.12,"176"],[1716557191.12,"176"],[1716559610.12,"176"],[1716562029.12,"176"],[1716564448.12,"181"],[1716566867.12,"176"],[1716569286.12,"176"],[1716571705.12,"176"],[1716574124.12,"301"],[1716576543.12,"176"],[1716578962.12,"176"],[1716581381.12,"176"],[1716583800.12,"176"],[1716586219.12,"177"]]}],"analysis":{},"seriesStatsCounter":{"Series":1869,"Chunks":27817,"Samples":3163325,"Bytes":3952496}}}

we can see the latter two returns very similar results

[christopherzli]

this is WIP

[christopherzli]

ready for review now

[hczhu-db]

Can we run some queries for which we know roughly how many time series/data samples are fetched as a basic correctness verification? Perhaps, use count(metric{}) to estimate the number of fetched time series.
Please paste the results in the PR description.

[hczhu-db]

@christopherzli Can you also explain your approach in the PR description?

[christopherzli]

Can we run some queries for which we know roughly how many time series/data samples are fetched as a basic correctness verification? Perhaps, use count(metric{}) to estimate the number of fetched time series.
Please paste the results in the PR description.

Please check the verification section. Count(up) is 173 and series fetched is 174.

want to call out if same series live in different store api(receive/obj storage), it will be counted separately as of now

[hczhu-db]

Can we run some queries for which we know roughly how many time series/data samples are fetched as a basic correctness verification? Perhaps, use count(metric{}) to estimate the number of fetched time series.
Please paste the results in the PR description.

Please check the verification section. Count(up) is 173 and series fetched is 174.

want to call out if same series live in different store api(receive/obj storage), it will be counted separately as of now

Can you run more queries (with range queries as well) to verify the results?

[jnyi]

this is great, I'd suggest to add some unit tests and communicate about this API change before merging it: https://github.com/thanos-io/thanos/pull/7390/files#r1615148051

[jnyi]

as we discussed, please work with @yuchen-db to make sure query bytes fetched show up in bronson UI in oregon-dev, thanks!

[christopherzli]

jenkins merge

will handle unit tests later