Skip to content

Commit

Permalink
migrate-group: Exclude S3 object keys containing "/" from syncing
Browse files Browse the repository at this point in the history
…even if they match *.json or *.md.  Instead, report them as unsynced
files.

This correctly handles an observed case of an object named
"archive/zika.json" that was dutifully copied over, though it would be
inaccessible.

Related-to: <nextstrain/private#66>
  • Loading branch information
tsibley committed Nov 16, 2022
1 parent 40ee390 commit c6802ae
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions scripts/migrate-group.js
Expand Up @@ -103,6 +103,7 @@ async function syncData({dryRun = true, group}) {
filters: [
"--exclude=*",
"--include=*.json",
"--exclude=*/*",
]
});

Expand All @@ -114,6 +115,7 @@ async function syncData({dryRun = true, group}) {
filters: [
"--exclude=*",
"--include=*.md",
"--exclude=*/*",
"--exclude=group-overview.md",
]
});
Expand All @@ -132,8 +134,8 @@ async function syncData({dryRun = true, group}) {

// Discover files to consider for manual review
const unsynced = (await s3ListObjects({group})).filter(
key => !key.endsWith(".json")
&& !key.endsWith(".md")
key => !(key.endsWith(".json") && !key.includes("/"))
&& !(key.endsWith(".md") && !key.includes("/"))
&& key !== "group-overview.md"
&& key !== "group-logo.png"
);
Expand Down

0 comments on commit c6802ae

Please sign in to comment.