Introduce Hardfork Generation Nightly Test

[dkijania]

Introducing Hardfork Generation Nightly test on fixed values:

NETWORK_NAME=devnet
CONFIG_JSON_GZ_URL=https://storage.googleapis.com/tmp-hardfork-testing/fork-devnet-march-22-2024.json.gz

In order to support this mechanism, I had to refactor a bit CI code. Extracted package generation job into separate and removed PackageGeneration Mode which was IMHO a duplication for Stable one. As a consequence I introduced new tag and filter to only run Hardfork pacakge generation jobs.

We need to change also pipeline setup for manually triggered jobs:

steps:
  - commands:
      - "dhall-to-yaml --quoted <<< './buildkite/src/Prepare.dhall' | buildkite-agent pipeline upload"
    label: ":pipeline:"
    agents:
       size: "generic"
    plugins:
      "docker#v3.5.0":
        environment:
          - BUILDKITE_AGENT_ACCESS_TOKEN
          - "BUILDKITE_PIPELINE_MODE=Stable"
          - "BUILDKITE_PIPELINE_STAGE=Generation"
          - "BUILDKITE_PIPELINE_FILTER=HardforkPackageGeneration"
        image: codaprotocol/ci-toolchain-base:v3
        mount-buildkite-agent: true
        propagate-environment: true

The last change is to update our nightly scheduled config and github webhook to add above two variables

[dkijania]

!ci-nightly-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

Nightly run : https://buildkite.com/o-1-labs-2/nightly-with-hf-package-generation-tests/builds/2

Manual run: https://buildkite.com/o-1-labs-2/mina-hardfork-package-generation-test-pipeline/builds/6

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

!ci-nightly-me

[dkijania]

!ci-build-me

[dkijania]

!ci-nightly-me

[dkijania]

!ci-build-me

[dkijania]

!ci-build-me

[dkijania]

Nightly run is green regarding hardfork jobs:

https://buildkite.com/o-1-labs-2/mina-end-to-end-nightlies/builds/1892#018f10d7-b5d8-4b59-977b-f407324cddc8

[dkijania]

!ci-build-me

[nholland94]

There's a TODO here after we ship berkeley that we need to also update the hardfork pipeline config in buildkite to correctly use the new way of selecting the hardfork jobs.