service to upload and retrieve latest files from Azure Blob Storage
The azure-storage package requires the environment variable AZURE_STORAGE_CONNECTION_STRING to be populated.
Instantiated using new AzureDataService(config).
The config object requires container, log, outputFile, outputDir, and version populated.
The summaryFile defaults to summary if none is provided.
Sample instantiation:
const version = '0.1';
const outputDir = './test/output';
const outputFile = 'test-data';
const summaryFilename = 'summary';
const containerName = 'data-test';
const log = logger;
const azureDataService = new AzureDataService({
containerName,
outputFile,
log,
outputDir,
summaryFilename,
version,
});
getLatestIds: downloads the latest datestamped seed IDs file from the specified Azure Storage location to the outputDir.
The Seed IDs filename is the output filename with a seed-ids suffix, i.e. in the example above the name would be test-data-seed-ids.
getLatestData: downloads the latest datestamped file matching the provided version and outputFile from the specified Azure Storage location to the outputDir.
All upload functions take a startMoment parameter used to datestamp the file. For more on the moment library, see the moment documentation.
uploadData uploads the file specified in outputFile from outputDir to Azure Storage, along with a datestamped and versioned copy.
uploadIds uploads the ID file from outputDir to seedIdFile in Azure Storage with datestamped suffix.
uploadSummary uploads the summary file from outputDir to Azure Storage, with an outputFile prefix, a datestamp, and a version suffix.
pruneFilesOlderThan removes all files prior to the provided moment parameter. The latest data, ID list and summary file will remain to prevent data loss.