Conversation
gcs_to_sftp example was missing setup and teardown stages after migration to the new design. Change-Id: I68e29f49fb4e1ac69f419c70a9ac46d8b8442e3b
| task_id="create_bucket", bucket_name=BUCKET_NAME, project_id=PROJECT_ID | ||
| ) | ||
|
|
||
| upload_file_1 = LocalFilesystemToGCSOperator( |
There was a problem hiding this comment.
wondering why you are uploading multiple times
There was a problem hiding this comment.
To ensure I have file available on different paths - directly in bucket root, under directory etc. It's later used by transfer operator and we are testing if its possible to copy file, copy file from dir (and recreate dir structures), copy with wildcard. But thanks to your question I have noticed we dont need third upload - since 2nd upload already puts file under dir and we came reuse it.
There was a problem hiding this comment.
I realized one of the operators has a "move_object" parameter which will delete files from the source directory. That's why we need all three separate uploads to cover all test scenarios:
Copy single file - 1st upload
Move file from the directory (removing source) - 2nd upload
Copy file from the directory when preserving dir structure - 3rd upload
Copy file from the directory without preserving dir structure - 3rd upload
3 participants
gcs_to_sftpexample was missing setup and teardown stages aftermigration to the new design. This PR adds them - now we upload file to GCS bucket before running transfer operators and remove GCS bucket at the end of the test.