Make automodel and flow queries work without submodule.

[starcke]

Resolve automodel and flow queries without submodule.

Checklist

• CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
• Issues have been created for any UI or other user-facing changes made by this pull request.
• [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

[starcke]

Unfortunately it seems I had some issues with this:

• It turned out that it only worked because I had some pre-downloaded dependencies in my CodeQL cache. Therefore we need to add an explicit pack download to make it works from clean slate.
• The createLockFileForStandardQuery was actually not as benign as we originally thought. It turns out it actually puts a lock file inside the standard package directory (e.g. /home/starcke/.codeql/packages/codeql/java-all). What is worse is if there is a dependency problem then it might put a partial lock file in there effectively corrupting the package (with no automatic recovery).

I believe I have fixed both of these problems:

• We explicitly download the needed pack (queries in addition to all).
• We only call createLockFileForStandardQuery in the fetch case, as that needs it to resolve the query.

I think the solution to the second problem is not the most elegant one. I think the best would probably be to stop using createLockFileForStandardQuery completely as that is dangerous to have around. One direction we could take here is to actually move the fetch queries to the codeql repo (which we also have other reasons for doing).

[aeisenberg]

So, this is creating a lock file in the downloaded library pack? It's generally something we should be avoiding since downloaded packs should be read-only. It looks like you have a cleanup job running that deletes the lock file, so that's a bit better. One concern I have is that you're likely to get different lock files over time as we release new transitive dependencies.

Moving the queries to their own query pack would be a better approach, in my view.

[starcke]

So, this is creating a lock file in the downloaded library pack? It's generally something we should be avoiding since downloaded packs should be read-only. It looks like you have a cleanup job running that deletes the lock file, so that's a bit better. One concern I have is that you're likely to get different lock files over time as we release new transitive dependencies.

It was doing that for the automodel queries (after I integrated them), and that was actually what corrupted my pack directory (as per our discussion on slack) because it failed half way through creating a broken lock file. I have now removed that lock creation from the automodel and flow queries. It is now only used by the fetch queries which need it (and here the lock creation happens in a temp dir), but longer term I would prefer to remove the creation of the lock completely as it feels very risky to use.

[charisk]

Looks good and works on my machine.