Skip to content

input.list: restore s3 manifest feature#39

Merged
tommyblue merged 1 commit intomainfrom
restore_list_s3_manifest
Sep 21, 2020
Merged

input.list: restore s3 manifest feature#39
tommyblue merged 1 commit intomainfrom
restore_list_s3_manifest

Conversation

@tommyblue
Copy link
Copy Markdown
Contributor

@tommyblue tommyblue commented Sep 17, 2020

❓ What

#35 removed an expected functionality of the List input: consuming a S3 manifest file containing a list of files to download and ingest.
This PR restores it back

✅ Checklists

  • Is there unit/integration test coverage for all new and/or changed functionality added in this PR?
  • Have the changes in this PR been functionally tested?
  • Has make gofmt-write been run on the code?
  • Has make govet been run on the code? Has the code been fixed accordingly to the output?
  • Have the changes been added to the CHANGELOG.md file?
  • Have the steps in CONTRIBUTING.md been followed to update a Go module?

@tommyblue tommyblue force-pushed the restore_list_s3_manifest branch 3 times, most recently from 7e0c139 to dd1b342 Compare September 17, 2020 13:56
@tommyblue tommyblue force-pushed the restore_list_s3_manifest branch from dd1b342 to e314f16 Compare September 17, 2020 13:57
Copy link
Copy Markdown
Collaborator

@arl arl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a nit. rest LGTM

Comment thread input/list.go
Comment on lines +339 to +350
} else {
resp, err := s.svc.GetObject(&s3.GetObjectInput{
Bucket: aws.String(u.Host),
Key: aws.String(u.Path),
})
if err != nil {
return err
}

s.processListFile(resp.Body)
resp.Body.Close()
return nil
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: better to return at the end of the if so as to avoid the else and un-indent the highlighted code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arl I did with the else (that I generally avoid) because the end of the if is unreachable (all the returns are inside the for loop. So I thought that, in this case, the if/else is a bit clearer

@tommyblue tommyblue merged commit 16aa6c6 into main Sep 21, 2020
@tommyblue tommyblue deleted the restore_list_s3_manifest branch September 21, 2020 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants