It's german. It means "the dev, the" #3212

zaneselvans · 2024-01-04T05:15:43Z

Overview

https://www.youtube.com/watch?v=gaXigSu72A4

Covers most of the tasks in #3179 which is the last bit of #3140

Update docs to refer to main, nightly, and stable branches rather than dev in explaining our git workflow and nightly build / data distribution setup.
Change all direct links to Read The Docs to use the newly default nightly docs rather than dev.
Update nightly build scripts and other github actions workflows to run off of main rather than dev.
Update dependabot and pre-commit-ci to run against main instead of dev.
Switch to using gs://builds.catalyst.coop for our build outputs.
Remove obsolete docker/.env file and define those envvars either in the container / workflow / script.

Testing

Manually ran a build off of this branch.

To-do list

Give feedback

Update the [release notes](../docs/release_notes.rst): reference the PR and related issues.

Update the release notes: reference the PR and related issues.
Options
Successfully updated the issue's project

There was an error updating the issue's project
Review the PR yourself and call out any questions or issues you have

Review the PR yourself and call out any questions or issues you have
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

.github/workflows/build-deploy-pudl.yml

zaneselvans · 2024-01-04T05:27:23Z

.github/workflows/build-deploy-pudl.yml

@@ -22,17 +21,15 @@ jobs:
      contents: write
      id-token: write
    steps:
-      - name: Use pudl-deployment-dev vm and dev branch if running on a schedule
+      - name: Use pudl-deployment-dev vm if running on a schedule
        if: ${{ (github.event_name == 'schedule') }}
        run: |
          echo "This action was triggered by a schedule."
          echo "GCE_INSTANCE=pudl-deployment-dev" >> $GITHUB_ENV


Hopefully we'll be on Batch shortly and the archaic naming of this VM won't matter.

I also think, at some point, we just trigger the whole build-deploy only on tag push and workflow dispatch, and just make the nightly tag on a schedule. Which will mean we don't have to do all these conditionals on the event name.

I've been thinking it might be nice to completely separate the build and deploy processes.

Like the build runs the ETL and tests / validations and fills up a directory in gs://builds.catalyst.coop as it does now, and then a totally separate script could be used to update the nightly branch, and deploy the outputs from a selected BUILD_ID to any combination of:

Datasette on Fly.io

AWS Open Data Registry S3 buckets

Our GCS buckets that mirror AWS

...

README.rst

docker/.env

docs/index.rst

docs/data_access.rst

docs/dev/nightly_data_builds.rst

docs/dev/project_management.rst

jdangerx

This looks great!

A few loose ends here and there, which I guess are technically not blocking the deprecation of dev. But, since we still also need to communicate to people about the new process, I think we're not in such a hurry. So maybe it's worth just fixing them.

In terms of communication, I think what we need to communicate is:

"New process, same as the old process, except now you branch off of and merge into main. Which means you don't have to manually change the base branch each time you make a PR, yay! Also don't mess with nightly/stable and still don't push things willy-nilly to main."

If that sounds good to you, feel free to copy-paste that right into Slack, hah.

.github/workflows/build-deploy-pudl.yml

jdangerx · 2024-01-04T16:29:18Z

.github/workflows/build-deploy-pudl.yml

@@ -22,17 +21,15 @@ jobs:
      contents: write
      id-token: write
    steps:
-      - name: Use pudl-deployment-dev vm and dev branch if running on a schedule
+      - name: Use pudl-deployment-dev vm if running on a schedule
        if: ${{ (github.event_name == 'schedule') }}
        run: |
          echo "This action was triggered by a schedule."
          echo "GCE_INSTANCE=pudl-deployment-dev" >> $GITHUB_ENV


I also think, at some point, we just trigger the whole build-deploy only on tag push and workflow dispatch, and just make the nightly tag on a schedule. Which will mean we don't have to do all these conditionals on the event name.

.github/workflows/update-conda-lockfile.yml

README.rst

docker/gcp_pudl_etl.sh

docker/vm_startup_script.sh

docs/data_access.rst

docs/dev/nightly_data_builds.rst

docs/dev/project_management.rst

.github/workflows/update-conda-lockfile.yml

README.rst

docker/.env

docker/vm_startup_script.sh

docs/dev/nightly_data_builds.rst

docs/dev/project_management.rst

zaneselvans · 2024-01-04T22:14:23Z

Hmm @bendnorman @jdangerx it looks like Google was really hoping for that .env file. Or is this a setting that's associated with the VM somewhere?

Updated [https://www.googleapis.com/compute/v1/projects/***/zones/***/instances/pudl-deployment-tag].
ERROR: (gcloud.compute.instances.update-container) Unable to read file [./docker/.env]: [Errno 2] No such file or directory: './docker/.env'
Error: Process completed with exit code 1.

Edit: ah nevermind we were specifying it in the deploy workflow.

- Update docs and build scripts to reflect new branch structure. - Run builds every night, but only if there are changes. - Rename nightly-build-outputs.catalyst.coop bucket to builds.catalyst.coop. - Remove docker/.env file and define those envvars elsewhere. - Add some basic release notes about our new branches / workflow

zaneselvans linked an issue Jan 4, 2024 that may be closed by this pull request

Deprecate and remove dev branch #3179

Closed

zaneselvans self-assigned this Jan 4, 2024

zaneselvans linked an issue Jan 4, 2024 that may be closed by this pull request

Move to trunk-based development #3140

Closed

zaneselvans commented Jan 4, 2024

View reviewed changes

zaneselvans requested a review from jdangerx January 4, 2024 05:41

jdangerx requested changes Jan 4, 2024

View reviewed changes

bendnorman reviewed Jan 4, 2024

View reviewed changes

jdangerx approved these changes Jan 4, 2024

View reviewed changes

bendnorman approved these changes Jan 4, 2024

View reviewed changes

zaneselvans force-pushed the die-dev-die branch from ea3be0d to e9c205a Compare January 5, 2024 03:57

zaneselvans merged commit e9c205a into dev Jan 5, 2024
10 of 11 checks passed

zaneselvans deleted the die-dev-die branch January 5, 2024 03:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

It's german. It means "the dev, the" #3212

It's german. It means "the dev, the" #3212

zaneselvans commented Jan 4, 2024 •

edited

To-do list

zaneselvans Jan 4, 2024

jdangerx Jan 4, 2024

zaneselvans Jan 4, 2024

jdangerx left a comment •

edited

jdangerx Jan 4, 2024

zaneselvans commented Jan 4, 2024 •

edited

It's german. It means "the dev, the" #3212

It's german. It means "the dev, the" #3212

Conversation

zaneselvans commented Jan 4, 2024 • edited

Overview

Testing

To-do list

zaneselvans Jan 4, 2024

Choose a reason for hiding this comment

jdangerx Jan 4, 2024

Choose a reason for hiding this comment

zaneselvans Jan 4, 2024

Choose a reason for hiding this comment

jdangerx left a comment • edited

Choose a reason for hiding this comment

jdangerx Jan 4, 2024

Choose a reason for hiding this comment

zaneselvans commented Jan 4, 2024 • edited

zaneselvans commented Jan 4, 2024 •

edited

jdangerx left a comment •

edited

zaneselvans commented Jan 4, 2024 •

edited