radar-output-restructure version 1.1.2

Changes since version 1.1.1:

• Tested and updated Azure storage
• Added Azure configuration example
• Optimized imports

radar-output-restructure version 1.1.1

Changes since version 1.1.0:

• Added logging messages on why a file cannot be cleaned
• Only extract files if they have reached a certain age, controlled by the worker.minimumFileAge configuration property.

radar-output-restructure version 1.1.0

Changes since version 1.0.1

• Fixed S3 directory listing and file name parsing. Added by #73 and #74
• Added a cleaner process that does an extraction validation of old files and remove them if valid. Added by #71
• Increased coverage of integration tests

radar-output-restructure version 1.0.1

Changes from Release 1.0.0

• Allows endOffset of filenames to be optional
  Closes #67
• Create target storage factory only once. Fixes #66.
• Adds configurable batch size

radar-output-restructure version 1.0.0

Changes since radar-hdfs-restructure version 0.6.0:

• Added storage drivers for S3 API input and output
• Storage is more consistent to configure
• Changed synchronisation and accounting to be handled with Redis
  • With more consistent locking
  • Faster write times
  • Redis is required.
• Added integration tests
• Configurable automatic deletion after a threshold amount of time, to avoid storage filling up.
• Reprocesses files that were modified after the last processed modification time.
• Customizable time bin format

radar-hdfs-restructure version 0.6.0

Changes since version 0.5.7:

• A new configuration file has been introduced. This allows for easier and more flexible configuration of the converter.
• Fixes an issue when using the output generator as a one-time application instead of a service.
• Converted Java code to Kotlin
• Refactored package names, including change to org.radarbase
• Track offsets in a per-topic offsets file in the offsets directory.
• Added an S3 storage driver.
• Use per-topic locking, to allow multiple restructure services to run simultaneously.
• Simplified parallelism.

Upgrade instructions:

• Write configuration file to match settings used with 0.5.7
• If needed, move all entries of offsets.csv to their own file in offsets/<topic>.csv. First go to the output directory, then run the bin/migrate-offsets-to-0.6.0.sh script.

radar-hdfs-restructure version 0.5.7

Changes since version 0.5.6:

• Run the restructure script as a service
• Updated gradle

radar-hdfs-restructure version 0.5.6

Changes since version 0.5.5:

• Corrects snappy decompression (fixes #43)

radar-hfds-restructure version 0.5.5

Changes since version 0.5.4:

• Added per-month directory key path factory org.radarcns.hdfs.MonthlyObservationKeyPathFactory
• Added --exclude-topic option to exclude certain topics from extraction
• Added ZIP compression
• Fixed integer overflow in record counts
• Updated unit testing to JUnit 5
• Updated Gradle

radar-hdfs-restructure version 0.5.4

Changes since version 0.5.3:

• Added setting for maximum number of files to process per topic --max-files-per-topic