Skip to content

StreamWriter should not require destination as constructor argument #88

@kevinwallimann

Description

@kevinwallimann

Description
Currently, the StreamWriter requires a destination as a constructor argument. However, e.g. writing to Kafka does not require a destination, but rather a topic.

Tasks

  • Remove the constructor argument destination from StreamWriter
  • Remove the default implementation of StreamWriter.getDestination and implement it in the derived classes.
  • Update archetype

Consequences

  • Currently, the destination directory is removed if the ingestion fails and the directory has been empty before the ingestion. This functionality assumes that the getDestination method of the StreamWriter returns a path on hdfs. However, this cannot be guaranteed. getDestination may return any string and it can't be assumed that it signifies a path.
    The functionality is not very useful in practice since the destination folder is empty only at the very first ingestion. Therefore, this functionality will be removed.

How to migrate

  • External components implementing StreamWriter need to remove the destination parameter from the constructor to the superclass, i.e. replace extends StreamWriter(destination) with extends StreamWriter
  • No configuration properties need to be changed

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions