fix(s3): Split bucket name and key before uploading #48
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
S3 files were uploaded to incorrect keys when running Airflow 2. This
was caused by differences between Airflow 1.10 and Airflow 2: the
latter assumes that when passing both bucket_name and key that the key
is to be taken as it is, where as the former seemed to work even when
the key contained the full url.
Now, we do the parsing and splitting ourselves. We would just set
bucket_name to None, however with Airflow 2.0 the upload function
checks for the presence of the bucket_name argument, to decide whether
to parse the url, not if it's set to None (I think this may be a bug).