Adding Pinot minion segment generation and push task.

[xiangfu0]

Description

Adding Pinot minion segment generation and push task.

Below is a table conf example with the simple version of batch ingestion config. More configs can be found in PR #6332.

{
  "tableName": "baseballStats",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "schemaName": "baseball",
    "replication": "1"
  },
  "tenants": {
  },
  "tableIndexConfig": {
    "loadMode": "HEAP",
    "invertedIndexColumns": [
      "playerID",
      "teamID"
    ]
  },
  "metadata": {
    "customConfigs": {
    }
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY",
      "batchConfigMaps": [
        {
          "inputDirURI": "s3://<my-bucket>/batch/baseballStats/rawdata",
          "includeFileNamePattern": "glob:**/*.csv",
          "excludeFileNamePattern": "glob:**/*.tmp",
          "inputFormat": "csv",
          "outputDirURI": "s3://<my-bucket>/batch/baseballStats/segments",
          "push.mode": "metadata"
        }
      ],
      "segmentNameSpec": {},
      "pushSpec": {}
    }
  },
  "task": {
    "taskTypeConfigsMap": {
      "SegmentGenerationAndPushTask": {
      }
    }
  }
}

For local test, the bare minimum configs are just inputDirURI and inputFormat:

{
  "tableName": "baseballStats",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "schemaName": "baseball",
    "replication": "1"
  },
  "tenants": {
  },
  "tableIndexConfig": {
    "loadMode": "HEAP",
    "invertedIndexColumns": [
      "playerID",
      "teamID"
    ]
  },
  "metadata": {
    "customConfigs": {
    }
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY",
      "batchConfigMaps": [
        {
          "inputDirURI": "examples/minions/batch/baseballStats/rawdata",
          "inputFormat": "csv"
        }
      ],
      "segmentNameSpec": {},
      "pushSpec": {}
    }
  },
  "task": {
    "taskTypeConfigsMap": {
      "SegmentGenerationAndPushTask": {
      }
    }
  }
}

[codecov-io]

Codecov Report

Merging #6340 (3828dd5) into master (1beaab5) will decrease coverage by 1.31%.
The diff coverage is 56.80%.

@@            Coverage Diff             @@
##           master    #6340      +/-   ##
==========================================
- Coverage   66.44%   65.13%   -1.32%     
==========================================
  Files        1075     1318     +243     
  Lines       54773    63979    +9206     
  Branches     8168     9299    +1131     
==========================================
+ Hits        36396    41671    +5275     
- Misses      15700    19341    +3641     
- Partials     2677     2967     +290     

Flag	Coverage Δ
unittests	65.13% <56.80%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...e/pinot/broker/api/resources/PinotBrokerDebug.java	0.00% <0.00%> (-79.32%)	⬇️
...ot/broker/broker/AllowAllAccessControlFactory.java	71.42% <ø> (-28.58%)	⬇️
.../helix/BrokerUserDefinedMessageHandlerFactory.java	33.96% <0.00%> (-32.71%)	⬇️
...ker/routing/instanceselector/InstanceSelector.java	100.00% <ø> (ø)
...ava/org/apache/pinot/client/AbstractResultSet.java	66.66% <0.00%> (+9.52%)	⬆️
.../main/java/org/apache/pinot/client/Connection.java	35.55% <0.00%> (-13.29%)	⬇️
...inot/client/JsonAsyncHttpPinotClientTransport.java	10.90% <0.00%> (-51.10%)	⬇️
...not/common/assignment/InstancePartitionsUtils.java	73.80% <ø> (+0.63%)	⬆️
...common/config/tuner/NoOpTableTableConfigTuner.java	100.00% <ø> (ø)
...ot/common/config/tuner/RealTimeAutoIndexTuner.java	100.00% <ø> (ø)
... and 1140 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea0bfa0...3828dd5. Read the comment docs.

[xiangfu0]

@laxmanchekka this is the PR of batch ingestion minion task. Just fyi.

[npawar]

From the code I see that the bare minimum config that has to be present in the batch config map is inputDirURI, inputFormat, outputDirURI. Is it possible to make outputDirURI also optional?

[xiangfu0]

From the code I see that the bare minimum config that has to be present in the batch config map is inputDirURI, inputFormat, outputDirURI. Is it possible to make outputDirURI also optional?

hmm, I think it's possible, we can just rely on the temp directory for segment build and then do tar push.

[xiangfu0]

From the code I see that the bare minimum config that has to be present in the batch config map is inputDirURI, inputFormat, outputDirURI. Is it possible to make outputDirURI also optional?

Updated the code to allow outputDirUri optional. See PR description for sample config