Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark sstfile generator #420

Merged
merged 26 commits into from Jun 27, 2019

Conversation

@spacewalkman
Copy link
Collaborator

commented May 22, 2019

Reopen a new PR after this repo changes from private to public, replacing PR#208

A spark job which does the following things:

parsing an input mapping file to map a hive table to a tag/edge, in which the table's PK(logically) should be identified
use nebula native client to encode a tag's key and values
define a custom hadoop OutputFormat and RecordWriter, which should generate a sub dir for one partition per worker in specified sst file output dir

@nebula-community-bot

This comment has been minimized.

Copy link

commented May 22, 2019

Can one of the admins verify this patch?

@sherman-the-tank

This comment has been minimized.

Copy link
Member

commented May 22, 2019

jenkins go

@nebula-community-bot

This comment has been minimized.

Copy link

commented May 22, 2019

Unit testing failed.

@spacewalkman

This comment has been minimized.

Copy link
Collaborator Author

commented May 23, 2019

CI failure seems to related to JNI header, repushed please let jenkins go

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

Jenkins go

@nebula-community-bot

This comment has been minimized.

Copy link

commented May 23, 2019

Unit testing failed.

@spacewalkman spacewalkman force-pushed the spacewalkman:spark-sstfile-generator branch 3 times, most recently from bf1f3d8 to 8f663e9 May 23, 2019

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented May 24, 2019

Jenkins go

@nebula-community-bot

This comment has been minimized.

Copy link

commented May 24, 2019

Unit testing failed.

@spacewalkman spacewalkman force-pushed the spacewalkman:spark-sstfile-generator branch 2 times, most recently from 1b2041d to 3c6955a May 24, 2019

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

Jenkins go

@dangleptr dangleptr requested review from darionyaphet and dangleptr May 27, 2019

@nebula-community-bot

This comment has been minimized.

Copy link

commented May 27, 2019

Unit testing failed.

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented May 29, 2019

Is the pr ready now? @spacewalkman

@spacewalkman

This comment has been minimized.

Copy link
Collaborator Author

commented May 29, 2019

@dangleptr There are some specific data skewness problem causing OOM, need to analysis input data.

@spacewalkman spacewalkman force-pushed the spacewalkman:spark-sstfile-generator branch from 3c6955a to 6af86d4 May 31, 2019

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2019

The pr is ready now? @spacewalkman

@spacewalkman

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 16, 2019

Yes.It's ready now.

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2019

Jenkins go

@nebula-community-bot

This comment has been minimized.

Copy link

commented Jun 17, 2019

Unit testing failed.

@dangleptr

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2019

Jenkins go

@nebula-community-bot

This comment has been minimized.

Copy link

commented Jun 17, 2019

Unit testing failed.

@spacewalkman spacewalkman force-pushed the spacewalkman:spark-sstfile-generator branch from 6af86d4 to 78cd2b3 Jun 19, 2019

@spacewalkman spacewalkman dismissed stale reviews from dutor and dangleptr via a63abbf Jun 27, 2019

@spacewalkman spacewalkman force-pushed the spacewalkman:spark-sstfile-generator branch from bc677f7 to a63abbf Jun 27, 2019

@spacewalkman

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 27, 2019

Jenkins, go

@nebula-community-bot

This comment has been minimized.

Copy link

commented Jun 27, 2019

Unit testing passed.

@nebula-community-bot

This comment has been minimized.

Copy link

commented Jun 27, 2019

Unit testing passed.

@nebula-community-bot

This comment has been minimized.

Copy link

commented Jun 27, 2019

Unit testing passed.

@dangleptr dangleptr merged commit 34eb36d into vesoft-inc:master Jun 27, 2019

1 check passed

UnitTest All tests passed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
7 participants
You can’t perform that action at this time.