Conversation
| } | ||
|
|
||
| func AddDirectoryPrefixFlag(cmd *cobra.Command) { | ||
| cmd.Flags().StringP("directory-prefix", "P", ".", "Set directory prefix where file will be saved") |
There was a problem hiding this comment.
I think if I wouldn't already know, I would not be sure what this means. Maybe something like output-directory would be clearer?
There was a problem hiding this comment.
Doesn't the description explain well?
(I'm biased since I sometimes use wget with -P option)
| } | ||
|
|
||
| // update the database after successful download | ||
| err = db.SetDataDump(path, filepath.Join(directoryPrefix, output)) |
There was a problem hiding this comment.
We're doing this even if --follow wasn't used, right? Is that what you intended?
There was a problem hiding this comment.
Yes it's intended since I think it's smart to not re-download already downloaded file with the follow option.
pro datadump download hours/api/20260115/20260115-00.gz
# skip 20260115-00.gz and download remains
pro datadump download hours/api/20260115/ --follow
cmd/pro/datadump/download.go
Outdated
| if err != nil { | ||
| return nil, fmt.Errorf("failed to check download status for %s: %w", file.Path, err) | ||
| } | ||
| // if force is set, re-download all files |
There was a problem hiding this comment.
Is there any legitimate reason why someone might want to use --follow --force? It will redownload all files and overwrite them if they already exist, right?
There was a problem hiding this comment.
To be honest, I don't know. But I think force re-downloading files if --follow --force is a right move when they are given.
| } | ||
|
|
||
| if !fileExists(localPath) { | ||
| // if file is deleted, remove it from the database |
There was a problem hiding this comment.
Why do we want that? I imagine what many people will do is have a script that uses --follow, then somehow ingest the downloaded data into their own system, then delete the downloaded files, and then later run again with --follow. The logic here would make it so that it will redownload the file that were already downloaded before. I'm not sure that is what people will expect. I would have assumed files that have been downloaded already will be skipped (even if they don't exist anymore). 🤔
There was a problem hiding this comment.
I intent to cover the following cases:
- A user deletes a file accidentally
- A user deletes a file intentionally and want to fetch it again afterwards
Co-authored-by: Florian Weingarten <fwgarten@gmail.com>
Co-authored-by: Florian Weingarten <fwgarten@gmail.com>
Co-authored-by: Florian Weingarten <fwgarten@gmail.com>
Co-authored-by: Florian Weingarten <fwgarten@gmail.com>
2 participants
Add
--followoption in the data dump's download command (#101)This PR also includes the following changes:
--directory-prefix/-Poption (same as wget)Note: following without date param is not implemented in this PR (since this PR is big already)