Skip to content

Data and query generator bytes-column support#6235

Open
apucher wants to merge 3 commits intomasterfrom
data-and-query-gen-bytes-support
Open

Data and query generator bytes-column support#6235
apucher wants to merge 3 commits intomasterfrom
data-and-query-gen-bytes-support

Conversation

@apucher
Copy link
Copy Markdown
Contributor

@apucher apucher commented Nov 4, 2020

Description

Add support for raw BYTES columns to data and query generators to expand future testing capabilities. Furthermore, support multi-value columns in the data generator and increase control over dialect and query logging in the query runner.

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
No

Does this PR fix a zero-downtime upgrade introduced earlier?
No

Does this PR otherwise need attention when creating release notes? Things to consider:
No

@apucher apucher requested a review from xiangfu0 November 4, 2020 20:35
@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #6235 into master will decrease coverage by 20.07%.
The diff coverage is 48.64%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #6235       +/-   ##
===========================================
- Coverage   66.44%   46.37%   -20.08%     
===========================================
  Files        1075     1240      +165     
  Lines       54773    58940     +4167     
  Branches     8168     8737      +569     
===========================================
- Hits        36396    27331     -9065     
- Misses      15700    29367    +13667     
+ Partials     2677     2242      -435     
Flag Coverage Δ
integration 46.37% <48.64%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ot/broker/broker/AllowAllAccessControlFactory.java 100.00% <ø> (ø)
.../helix/BrokerUserDefinedMessageHandlerFactory.java 52.83% <0.00%> (-13.84%) ⬇️
...org/apache/pinot/broker/queryquota/HitCounter.java 0.00% <0.00%> (-100.00%) ⬇️
...che/pinot/broker/queryquota/MaxHitRateTracker.java 0.00% <0.00%> (ø)
...ache/pinot/broker/queryquota/QueryQuotaEntity.java 0.00% <0.00%> (-50.00%) ⬇️
...routing/instanceselector/BaseInstanceSelector.java 96.03% <ø> (-3.97%) ⬇️
...ava/org/apache/pinot/client/AbstractResultSet.java 26.66% <0.00%> (-30.48%) ⬇️
.../main/java/org/apache/pinot/client/Connection.java 22.22% <0.00%> (-26.62%) ⬇️
.../org/apache/pinot/client/ResultTableResultSet.java 24.00% <0.00%> (-10.29%) ⬇️
...not/common/assignment/InstancePartitionsUtils.java 64.28% <ø> (-8.89%) ⬇️
... and 1229 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f4ee68...f87b940. Read the comment docs.

@apucher apucher requested a review from Jackie-Jiang November 11, 2020 20:17
Comment on lines +188 to +190
// for raw bytes
String hexRaw = StringUtils.stripStart(Hex.encodeHexString(((ByteBuffer) avroValue).array()), "0");
String hexAligned = (hexRaw.length() & 0x1) == 0 ? hexRaw : "0" + hexRaw;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not directly using Hex.encodeHexString(((ByteBuffer) avroValue).array())?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, does Avro always have ByteBuffer backed by an array?


private final Map<String, DataType> dataTypesMap;
private final Map<String, FieldType> fieldTypesMap;
private final Map<String, Boolean> singleValueFlagsMap;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we capable of generating MV values?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants