Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reduce bytearray copy to minimal optimize VSizeIndexedWriter #2079

Merged
merged 1 commit into from
Dec 11, 2015
Merged

reduce bytearray copy to minimal optimize VSizeIndexedWriter #2079

merged 1 commit into from
Dec 11, 2015

Conversation

binlijin
Copy link
Contributor

No description provided.

@binlijin binlijin closed this Dec 10, 2015
@binlijin binlijin reopened this Dec 10, 2015
xvrl
xvrl reviewed Dec 10, 2015
View reviewed changes
processing/src/main/java/io/druid/segment/data/VSizeIndexedInts.java
*/
public static byte[] getBytesNoPaddingfromList(List<Integer> list, int maxValue)
{
int numBytes = getNumBytesForMax(maxValue);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we refactor to reduce copy-paste with fromList?

@xvrl
Copy link
Member

xvrl commented Dec 10, 2015

@binlijin any numbers on how much this improves performance?

@binlijin
Copy link
Contributor Author

@xvrl , the performance numbers is：
Have 95 dimensions
Before：
round1
2015-12-10T15:49:19,781 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base8884394285534279712flush/merged/v8-tmp] walked 500,000/500,000 rows in 16,840 millis.
2015-12-10T15:49:40,627 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base8884394285534279712flush/merged/v8-tmp] completed walk through of 916,822 rows in 37,767 millis.
round2
2015-12-10T15:50:59,429 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base774856129942668621flush/merged/v8-tmp] walked 500,000/500,000 rows in 16,473 millis.
2015-12-10T15:51:19,962 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base774856129942668621flush/merged/v8-tmp] completed walk through of 916,822 rows in 37,065 millis.

After：
round1
2015-12-10T15:52:41,049 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4028347591001450805flush/merged/v8-tmp] walked 500,000/500,000 rows in 13,857 millis.
2015-12-10T15:52:58,643 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4028347591001450805flush/merged/v8-tmp] completed walk through of 916,822 rows in 31,505 millis.
round2
2015-12-10T15:54:14,273 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4543915257477602206flush/merged/v8-tmp] walked 500,000/500,000 rows in 14,390 millis.
2015-12-10T15:54:31,960 INFO [main] io.druid.segment.IndexMerger - outDir[/var/folders/z7/g5zy3kfj7t54y1f074hsqw7h0000gn/T/base4543915257477602206flush/merged/v8-tmp] completed walk through of 916,822 rows in 32,169 millis.

@binlijin binlijin closed this Dec 10, 2015
@binlijin binlijin reopened this Dec 10, 2015
@nishantmonu51
Copy link
Member

👍

xvrl added a commit that referenced this pull request Dec 11, 2015
@xvrl
Merge pull request #2079 from binlijin/master
d531e69 
reduce bytearray copy to minimal optimize VSizeIndexedWriter
@xvrl xvrl merged commit d531e69 into apache:master Dec 11, 2015
@fjy fjy modified the milestone: 0.9.0 Feb 4, 2016
@fjy fjy mentioned this pull request Feb 5, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.9.0
Development

Successfully merging this pull request may close these issues.
4 participants
@binlijin @xvrl @nishantmonu51 @fjy