remove Firehose and FirehoseFactory #16758
Conversation
changes: * removed `Firehose` and `FirehoseFactory` and remaining implementations which were mostly no longer used after apache#16602 * Moved `IngestSegmentFirehose` which was still used internally by Hadoop ingestion to `DatasourceRecordReader.SegmentReader` * Rename `SQLFirehoseFactoryDatabaseConnector` to `SQLInputSourceDatabaseConnector` and similar renames for sub-classes * Moved anything remaining in a 'firehose' package somewhere else * Clean up docs on firehose stuff
github-actions
bot
added
Area - Documentation
Area - Batch Ingestion
Area - Metrics/Event Emitting
Area - Segment Format and Ser/De
Area - Streaming Ingestion
Area - Dependencies
Kubernetes
Area - MSQ
There was a problem hiding this comment.
Thanks, @clintropolis !
I have left some minor suggestions, none of which are blockers for this PR.
With the exception of the renamed SegmentReader class, all the changes look good to me (I still need to go through SegmentReader).
| @JsonInclude(Include.NON_NULL) | ||
| @Deprecated | ||
| public FirehoseFactory getFirehoseFactory() | ||
| public IndexIOConfig(@Nullable Boolean appendToExisting, @Nullable Boolean dropExisting) |
There was a problem hiding this comment.
Do we still need this constructor?
There was a problem hiding this comment.
good catch, removed
| @@ -152,8 +152,7 @@ default int getPriority() | |||
| * the task does not use any. Users can be given permission to access particular types of | |||
| * input sources but not others, using the | |||
| * {@link org.apache.druid.server.security.AuthConfig#enableInputSourceSecurity} config. | |||
| * @throws UnsupportedOperationException if the given task type does not suppoert input source based security. Such | |||
| * would be the case, if the task uses firehose. | |||
| * @throws UnsupportedOperationException if the given task type does not suppoert input source based security | |||
There was a problem hiding this comment.
I guess now it will never throw this exception.
There was a problem hiding this comment.
the default implementation still throws that exception, so i suppose if something doesn't implement this method it would happen
| public ParallelIndexIOConfig(@Nullable Boolean appendToExisting) | ||
| { | ||
| this(firehoseFactory, null, null, appendToExisting, null); | ||
| this(null, null, appendToExisting, null); | ||
| } | ||
|
|
||
| @Deprecated | ||
| public ParallelIndexIOConfig(FirehoseFactory firehoseFactory, @Nullable Boolean appendToExisting, boolean dropExisting) | ||
| public ParallelIndexIOConfig(@Nullable Boolean appendToExisting, boolean dropExisting) |
There was a problem hiding this comment.
We can completely remove these constructors now.
| @@ -163,4 +189,157 @@ public void close() throws IOException | |||
| FileUtils.deleteDirectory(dir); | |||
| } | |||
| } | |||
|
|
|||
| public static class SegmentReader implements Closeable | |||
There was a problem hiding this comment.
I think it would be easier to review this if we leave this class in a separate file and just rename IngestSegmentFirehose to SegmentReader.
There was a problem hiding this comment.
left this as an internal class btw because its only used internally to DatasourceRecordReader
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| public void testConnectorValidationInvalidUri() | ||
| { | ||
| derbyConnector = derbyConnectorRule.getConnector(); | ||
| final List<String> sqls = SqlTestUtils.selectFrom(TABLE_1, TABLE_2); |
Check notice
Code scanning / CodeQL
Unread local variable Note
Mostly #16652 but without renaming
SinkandFireHydrant.changes:
FirehoseandFirehoseFactoryand remaining implementations which were mostly no longer used after Remove index_realtime and index_realtime_appenderator tasks #16602IngestSegmentFirehosewhich was still used internally by Hadoop ingestion toDatasourceRecordReader.SegmentReaderSQLFirehoseFactoryDatabaseConnectortoSQLInputSourceDatabaseConnectorand similar renames for sub-classes