Skip to content

CI: inline the move-caches script into the action #12213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 79 additions & 8 deletions .github/actions/cache-query-compilation/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,83 @@ runs:
codeql-compile-${{ inputs.key }}-${{ github.ref_name }}-
codeql-compile-${{ inputs.key }}-main-
- name: Fill compilation cache directory
id: fill-compilation-dir
shell: bash
run: |
# Move all the existing cache into another folder, so we only preserve the cache for the current queries.
node $GITHUB_WORKSPACE/.github/actions/cache-query-compilation/move-caches.js ${COMBINED_CACHE_DIR}

echo "compdir=${COMBINED_CACHE_DIR}" >> $GITHUB_OUTPUT
env:
uses: actions/github-script@v6
env:
COMBINED_CACHE_DIR: ${{ runner.temp }}/compilation-dir
with:
script: |
// # Move all the existing cache into another folder, so we only preserve the cache for the current queries.
// mkdir -p ${COMBINED_CACHE_DIR}
// rm -f **/.cache/{lock,size} # -f to avoid errors if the cache is empty.
// # copy the contents of the .cache folders into the combined cache folder.
// cp -r **/.cache/* ${COMBINED_CACHE_DIR}/ || : # ignore missing files
// # clean up the .cache folders
// rm -rf **/.cache/*
Comment on lines +51 to +57
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete?

Suggested change
// # Move all the existing cache into another folder, so we only preserve the cache for the current queries.
// mkdir -p ${COMBINED_CACHE_DIR}
// rm -f **/.cache/{lock,size} # -f to avoid errors if the cache is empty.
// # copy the contents of the .cache folders into the combined cache folder.
// cp -r **/.cache/* ${COMBINED_CACHE_DIR}/ || : # ignore missing files
// # clean up the .cache folders
// rm -rf **/.cache/*


const fs = require("fs");
const path = require("path");

// the first argv is the cache folder to create.
const COMBINED_CACHE_DIR = process.env.COMBINED_CACHE_DIR;

function* walkCaches(dir) {
const files = fs.readdirSync(dir, { withFileTypes: true });
for (const file of files) {
if (file.isDirectory()) {
const filePath = path.join(dir, file.name);
yield* walkCaches(filePath);
if (file.name === ".cache") {
yield filePath;
}
}
}
}

async function copyDir(src, dest) {
for await (const file of await fs.promises.readdir(src, { withFileTypes: true })) {
const srcPath = path.join(src, file.name);
const destPath = path.join(dest, file.name);
if (file.isDirectory()) {
if (!fs.existsSync(destPath)) {
fs.mkdirSync(destPath);
}
await copyDir(srcPath, destPath);
} else {
await fs.promises.copyFile(srcPath, destPath);
}
}
}

async function main() {
const cacheDirs = [...walkCaches(".")];

for (const dir of cacheDirs) {
console.log(`Found .cache dir at ${dir}`);
}

// mkdir -p ${COMBINED_CACHE_DIR}
fs.mkdirSync(COMBINED_CACHE_DIR, { recursive: true });

// rm -f **/.cache/{lock,size} # -f to avoid errors if the cache is empty.
await Promise.all(
cacheDirs.map((cacheDir) =>
(async function () {
await fs.promises.rm(path.join(cacheDir, "lock"), { force: true });
await fs.promises.rm(path.join(cacheDir, "size"), { force: true });
})()
)
);

// # copy the contents of the .cache folders into the combined cache folder.
// cp -r **/.cache/* ${COMBINED_CACHE_DIR}/ || : # ignore missing files
await Promise.all(
cacheDirs.map((cacheDir) => copyDir(cacheDir, COMBINED_CACHE_DIR))
);

// # clean up the .cache folders
// rm -rf **/.cache/*
await Promise.all(
cacheDirs.map((cacheDir) => fs.promises.rm(cacheDir, { recursive: true }))
);
}
main();
75 changes: 0 additions & 75 deletions .github/actions/cache-query-compilation/move-caches.js

This file was deleted.

7 changes: 0 additions & 7 deletions .github/workflows/ql-for-ql-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,6 @@ on:
branches: [main]
pull_request:
branches: [main]
paths:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove these paths?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to trigger QL-for-QL on this PR.
The simplest approach was just to always run QL-for-QL.

- "ql/**"
- "**.qll"
- "**.ql"
- "**.dbscheme"
- "**/qlpack.yml"
- ".github/workflows/ql-for-ql-build.yml"

env:
CARGO_TERM_COLOR: always
Expand Down