Enhance longrun output plot summary

[LenkaNovak]

build

https://buildkite.com/clima/climacoupler-longruns/builds/558

components

• rm slabplanet conservation logs
• NCEP and AMIP plots comparison on one page
• apply leadership board in AMIP longruns (path fixed and added seasonal)
• build_history as a direct link to Buildkite
• make all runs soft fail (here hard fails are not necessary and they can stop the slack report from other passing runs)
• add an array of dates to bias plots
• Error 26 (this only happened for a 2 hour period and worked after retry)
• rm scaling tests - these will be moved to a separate pipeline (on GPU) add GPU AMIP scaling runs #663
• included @Sbozzolo 's changes in Use output dates from atmos.integrator #725

Buildkite URL issue

• The URL link pass can work but it looks like there is a time limit for how long it is accessible:

      - label: ":envelope: Slack report: build_history new"
        command:
          - |
            ARTIFACT_ID=$(curl -s -H "Authorization: Bearer $BUILDKITE_READ_ARTIFACTS_API_TOKEN" -X GET "https://api.buildkite.com/v2/organizations/clima/pipelines/climacoupler-longruns/builds/$$BUILDKITE_BUILD_NUMBER/artifacts" | jq -r '.[] | select(.filename == "build_history.html") | .id')
            JOB_ID=$(curl -s -H "Authorization: Bearer $BUILDKITE_READ_ARTIFACTS_API_TOKEN" -X GET "https://api.buildkite.com/v2/organizations/clima/pipelines/climacoupler-longruns/builds/$$BUILDKITE_BUILD_NUMBER/artifacts" | jq -r '.[] | select(.filename == "build_history.html") | .job_id')
            URL=$(curl -H "Authorization: Bearer $BUILDKITE_READ_ARTIFACTS_API_TOKEN"   -X GET "https://api.buildkite.com/v2/organizations/clima/pipelines/climacoupler-longruns/builds/$$BUILDKITE_BUILD_NUMBER/jobs/$$JOB_ID/artifacts/$$ARTIFACT_ID/download"| jq -r '.url')
            echo "Interactive overall job performance history $$URL"
            slack-message -c "#coupler-report" -x ":rocket: Interactive overall job performance history <$${URL}|build_history> :rocket:"

• When viewing an artifact on Buildkite, a new URL is generated each time the artifact is being read, and each URL is only valid for 60s (see here). For a longer lasting URL, we'd need to set up an S3 bucket (see here) but it was decided this would not be worth the time investment.