handle different format of return values (#61) markdown formatting fixes (#54) Fusillade Authz helper function (#57)
dcplib.etl: use DSS GET bundles/all when no query is given (#60)
skip flaky test Add update date and submission errors to ingest entities (#59)
Update trufflehog regex. (#56) Update trufflehog patterns. (#55) add processes to ingest entities (#53)
include envelopes with submitted status as in process (#52)
- Reduce biomaterials end point page size to 20 for ingest api as larger page sizes lead to frequent timeouts (#51)
- Authn JWT helpers (#45)
- Close handle on ChecksummingBufferedReader exit
- Add test for empty bundles (#33)
- Metadata project_shortname has been replaced by project_short_name, make ingest_entities.Project work with both.
- Added IngestApiAgent and ingest_entities (SubmissionEnvelope etc...) which are used by data-monitoring-dashboard, dcp-diag and soon the dcp integration tests.
-
ETL: add page_processor and page_size args to extract function (#46)
-
Drop support for Python 3.4
- Drop support for Python 2.7, add 3.7
- dcpquery.etl.DSSExtractor.get_file is public
- ETL: make retry-after behavior configurable; improve logging (#36)
-
ETL: fetch all pages for GET bundle (#39)
-
AwsSecret: add debug prints; add methods exists_in_aws and is_deleted
-
Refactor ETL (DSSExtractor) (#28)
-
Test fixes
- Add SQS batch context manager (#26)
- Fix erroneous initialization of ETL DSS client
-
Fix redirect logic when Retry-After is given with a 301 redirect (#23)
-
ETL fixes: Configurable dispatch for empty bundles (#18); do not configure global logging
- Fix release
- Remove gpg signing from twine
- Added twine to requirements-dev
- Adding module for working with SQS. (#20)
- networking.HTTPRequest: set max redirects to a high number (#19)
- Add padding for crc32c calculation to make it 8 characters in length to be backwards compatible. (#17)
-
Removing crcmod dependency from dcplib (#16)
-
ETL: consume dispatch callbacks futures, add tests (#15)
- Add test helpers originally from DSS (#11)
- ETL: Pass bundle manifest to transformer (#14)
-
add ETL option to parallelize transformers and loaders (#13)
-
Make dcplib.etl.DSSExtractor picklable (#12)
-
Add trufflehog regexes to dcplib. (#10)
-
Add AWS utilities used by DSS and Query Service (#9)
-
Switch to the ICRAR implementation of CRC32C (#6)
-
Pin puremagic dependency to version 1.4
-
Ignore Python 3.6 code when testing on lower versions of Python
-
Fix directory initialization bug in ETL
-
Shared DSS client ETL code (#4)
-
Add dcplib.networking.http (#5)
- Config: allow customization of secret name
- S3Etag is now optionally configured with chunk size
- ChecksummingBufferedReader is now optionally configured with chunk size
- Add s3_multipart.MULTIPART_THRESHOLD and tests for get_s3_multipart_chunk_size()
- Add get_s3_multipart_chunk_size method
- Python 2.7 compatiility
- Add Config and AwsSecret classes
- Fold checksumming_io into dcplib.
- Retry.
- MediaType and DcpMediaType classes.