Description
Currently, the StreamWriter requires a destination as a constructor argument. However, e.g. writing to Kafka does not require a destination, but rather a topic.
Tasks
- Remove the constructor argument
destination from StreamWriter
- Remove the default implementation of
StreamWriter.getDestination and implement it in the derived classes.
- Update archetype
Consequences
- Currently, the destination directory is removed if the ingestion fails and the directory has been empty before the ingestion. This functionality assumes that the
getDestination method of the StreamWriter returns a path on hdfs. However, this cannot be guaranteed. getDestination may return any string and it can't be assumed that it signifies a path.
The functionality is not very useful in practice since the destination folder is empty only at the very first ingestion. Therefore, this functionality will be removed.
How to migrate
- External components implementing
StreamWriter need to remove the destination parameter from the constructor to the superclass, i.e. replace extends StreamWriter(destination) with extends StreamWriter
- No configuration properties need to be changed
Description
Currently, the
StreamWriterrequires a destination as a constructor argument. However, e.g. writing to Kafka does not require a destination, but rather a topic.Tasks
destinationfromStreamWriterStreamWriter.getDestinationand implement it in the derived classes.Consequences
getDestinationmethod of theStreamWriterreturns a path on hdfs. However, this cannot be guaranteed.getDestinationmay return any string and it can't be assumed that it signifies a path.The functionality is not very useful in practice since the destination folder is empty only at the very first ingestion. Therefore, this functionality will be removed.
How to migrate
StreamWriterneed to remove thedestinationparameter from the constructor to the superclass, i.e. replaceextends StreamWriter(destination)withextends StreamWriter