ci: implement queuing mechanism for acceptance tests

[lionel-nj]

Summary:

This PR provides support to implement a queueing mechanism dedicated to the acceptance test feature. Said mechanism has to be implemented since running the validator on 1000+ datasets sequentially takes more than 6 hours (on github) which causes the workflow to get cancelled

Expected behavior:

• CI should be triggered on all commits except if the developer added the keyword "[acceptance test skip]" to its latest commit message
• Older workflows (only Rule acceptance tests) should be cancelled when a new one is triggered to avoid queueing too many workflows for a same PR
• Datasets' URL should be fetched from the MobilityDatabase
• For each dataset (i.e. URL retrieved from the database), the reference version of the validator (from the master branch) should run and produce reference.json and reference_errors.json
• For each dataset (i.e. URL retrieved from the database), the snapshot version of the validator (from the branch where this CI process is run) should run and produce report.json and system_errors.json

⚙️ The task get-reports (from acceptance_test.yml) works as follows:
Once 1000+ urls are retrieved from the Mobility Database as a list of 256 objects (max), each one of these objects are used to define a matrix. The latest generates jobs that run concurrently. Each one of these jobs run queue_runner.sh which parses the input as an array of json objects:

({"id:"dataset id value-1", "url":"dataset url-1"} {"id:"dataset id value-2", "url":"dataset url-2"} ... {"id:"dataset id value-n", "url":"dataset url-n"})

For each one of these dictionaries both snapshot and reference validator are run.

The reports are all saved in a directory called output - which is persisted at the end of the job, and uploaded to Google Cloud Storage to ease download.

Please make sure these boxes are checked before submitting your pull request - thanks!

• Run the unit tests with gradle test to make sure you didn't break anything
• Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
• Linked all relevant issues
• Include screenshot(s) showing how this pull request works and fixes the issue(s)

[lionel-nj]

@barbeau, as demonstrated by 7fd762c (see https://github.com/MobilityData/gtfs-validator/runs/4010914933?check_suite_focus=true), including "[acceptance test skip]" in the commit message enables to skip the execution of Rule acceptance test workflow. When the keyword is not included in said commit message, the workflow is executed as shown in https://github.com/MobilityData/gtfs-validator/actions/runs/1386081082 from 7fd762c. PTAL.

Note: this mechanism will be documented in #848 where additional documentation for the acceptance test has been created.

[barbeau]

Great job @lionel-nj, LGTM!

[barbeau]

For future optimization - I'm not sure what the primary cost is for fetching data from the Mobility Database, but if it's outbound data bandwidth from the Mobility Database you could reduce this by supporting the HTTP header If-Modified-Since and caching a version of the response in the GitHub Action. Then when you send this request to the mobility database with the date of the cached file, you'd only get the full dataset returned to you (and pay for it) if it was modified after the date of the cache file. Both sides (GitHub Action and Mobility Database) would need to support If-Modified-Since for that to work.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since

[lionel-nj]

Thanks! I will definitely look into that!