Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding support for Protobuf input format #5293

Merged
merged 10 commits into from May 27, 2020
Merged

Adding support for Protobuf input format #5293

merged 10 commits into from May 27, 2020

Conversation

KKcorps
Copy link
Contributor

@KKcorps KKcorps commented Apr 23, 2020 •
edited

  • Adding Protobuf Record Extractor and Record Reader
  • Using File Descriptor to infer ProtoBuf Schema instead of Generated Class
  • Unit test included

Sample config for Ingestion

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: 'file:///path/to/input'
includeFileNamePattern: 'glob:**/*.parquet'
excludeFileNamePattern: 'glob:**/*.avro'
outputDirURI: 'file:///path/to/output'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'proto'
  className: 'org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReaderConfig'
  configs:
    descriptorFile: 'file:///path/to/sample.desc
tableSpec:
  tableName: 'myTable'
  schemaURI: 'http://localhost:9000/tables/myTable/schema'
  tableConfigURI: 'http://localhost:9000/tables/myTable'
pinotClusterSpecs:
  - controllerURI: 'localhost:9000'
pushJobSpec:
  pushAttempts: 2

@KKcorps
Copy link
Contributor Author

KKcorps commented Apr 23, 2020

Should I also make changes in org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper to pass custom Reader config as it is currently done for CSV and Thrift.

@mayankshriv
Copy link
Contributor

Should I also make changes in org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper to pass custom Reader config as it is currently done for CSV and Thrift.

Yes, that would be great.

@kishoreg
Copy link
Member

Should I also make changes in org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper to pass custom Reader config as it is currently done for CSV and Thrift.

I think that's deprecated and all configurations will go through SegmentGenerationJobSpec. https://github.com/apache/incubator-pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/spec/RecordReaderSpec.java

kishoreg
kishoreg requested changes Apr 26, 2020
View reviewed changes
Copy link
Member

@kishoreg kishoreg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments

...rotobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java Outdated Show resolved Hide resolved
...f/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReaderConfig.java Outdated Show resolved Hide resolved
@kishoreg
Copy link
Member

kishoreg commented May 9, 2020

@KKcorps is this ready to go?

@KKcorps
Copy link
Contributor Author

KKcorps commented May 9, 2020

@kishoreg Yes, it is complete from my end

npawar
npawar reviewed May 13, 2020
View reviewed changes
...rotobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java Outdated Show resolved Hide resolved
...rotobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java Outdated Show resolved Hide resolved
...rotobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java Outdated Show resolved Hide resolved
...rotobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java Outdated Show resolved Hide resolved
...rc/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordExtractorConfig.java Outdated Show resolved Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/utils/ResourceFinder.java Show resolved Hide resolved
pinot-plugins/pinot-input-format/pinot-protobuf/src/test/resources/sample.proto Show resolved Hide resolved
npawar
npawar approved these changes May 27, 2020
View reviewed changes
Copy link
Contributor

@npawar npawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just 1 comment. LGTM otherwise

kishoreg
kishoreg approved these changes May 27, 2020
View reviewed changes
Copy link
Member

@kishoreg kishoreg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! Thanks for wrapping this up

@npawar
Copy link
Contributor

npawar commented May 27, 2020

Linking to issue #5269

@npawar npawar merged commit b69453e into apache:master May 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kishoreg kishoreg kishoreg approved these changes

@npawar npawar npawar approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@KKcorps @mayankshriv @kishoreg @npawar