Add workflow to gen data #329
Open
+215
−72
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is an automated update by travis-ci at Sat Apr 13 17:13:22 PDT 2024 [skip ci]
ckingbailey
force-pushed
the
feat/pull-v2-api
branch
from
11fdf8e
to
5769d8b
Compare
This is an automated update by travis-ci at Tue May 28 21:29:40 PDT 2024 [skip ci]
Build diff from Commit 1e99f65:diff --git a/build/_data/elections/oakland/2018-06-05.json b/build/_data/elections/oakland/2018-06-05.json
index b39b338..140316d 100644
--- a/build/_data/elections/oakland/2018-06-05.json
+++ b/build/_data/elections/oakland/2018-06-05.json
@@ -57,7 +57,7 @@
"total_contributions": 15000.0
},
{
- "name": "Service Employees International Union Local 1021 Issues PAC",
+ "name": "Oakland Athletics Baseball Company",
"type": "Measure",
"election_name": "oakland-june-2018",
"total_contributions": 10000.0
diff --git a/build/_data/elections/oakland/2023-11-07.json b/build/_data/elections/oakland/2023-11-07.json
index 4e77c51..1e5045c 100644
--- a/build/_data/elections/oakland/2023-11-07.json
+++ b/build/_data/elections/oakland/2023-11-07.json
@@ -73,19 +73,19 @@
],
"top_contributors_for_offices": [
{
- "name": "UA Local 342",
+ "name": "Service Employees International Union Local 1021 Candidate PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Service Employees International Union Local 1021 Candidate PAC",
+ "name": "Oakland Education Association PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Peralta Federation of Teachers COPE",
+ "name": "Families in Action for Justice",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
diff --git a/build/_data/elections/oakland/2024-03-05.json b/build/_data/elections/oakland/2024-03-05.json
index db9ca88..a6261f2 100644
--- a/build/_data/elections/oakland/2024-03-05.json
+++ b/build/_data/elections/oakland/2024-03-05.json
@@ -48,13 +48,13 @@
],
"top_contributors_for_offices": [
{
- "name": "Steven Von Stade",
+ "name": "Monique C Houston",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
},
{
- "name": "Monique C Houston",
+ "name": "Steven Von Stade",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
diff --git a/build/_data/elections/oakland/2024-11-05.json b/build/_data/elections/oakland/2024-11-05.json
index 8440b5b..8bd5599 100644
--- a/build/_data/elections/oakland/2024-11-05.json
+++ b/build/_data/elections/oakland/2024-11-05.json
@@ -79,19 +79,19 @@
],
"top_contributors_for_offices": [
{
- "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+ "name": "International Association of Firefighters Local 55",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "International Association of Firefighters Local 55",
+ "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "Weylin White",
+ "name": "Todd Scanlin",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 1800.0
diff --git a/build/_data/stats.json b/build/_data/stats.json
index 74f4147..35aa4a8 100644
--- a/build/_data/stats.json
+++ b/build/_data/stats.json
@@ -1,3 +1,3 @@
{
- "date_processed": "2024-04-13 00:08:40 -0700"
+ "date_processed": "2024-05-29 00:08:29 -0700"
}
diff --git a/build/_data/totals.json b/build/_data/totals.json
index f6a1f4e..f82e1c3 100644
--- a/build/_data/totals.json
+++ b/build/_data/totals.json
@@ -1032,7 +1032,7 @@
"total_contributions": 15000.0
},
{
- "name": "Service Employees International Union Local 1021 Issues PAC",
+ "name": "Oakland Athletics Baseball Company",
"type": "Measure",
"election_name": "oakland-june-2018",
"total_contributions": 10000.0
@@ -1385,19 +1385,19 @@
],
"top_contributors_for_offices": [
{
- "name": "UA Local 342",
+ "name": "Service Employees International Union Local 1021 Candidate PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Service Employees International Union Local 1021 Candidate PAC",
+ "name": "Oakland Education Association PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Peralta Federation of Teachers COPE",
+ "name": "Families in Action for Justice",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
@@ -1500,13 +1500,13 @@
],
"top_contributors_for_offices": [
{
- "name": "Steven Von Stade",
+ "name": "Monique C Houston",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
},
{
- "name": "Monique C Houston",
+ "name": "Steven Von Stade",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
@@ -1646,19 +1646,19 @@
],
"top_contributors_for_offices": [
{
- "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+ "name": "International Association of Firefighters Local 55",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "International Association of Firefighters Local 55",
+ "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "Weylin White",
+ "name": "Todd Scanlin",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 1800.0 |
This is an automated update by travis-ci at Wed May 29 19:54:16 PDT 2024 [skip ci]
This is an automated update by travis-ci at Wed May 29 21:15:13 PDT 2024 [skip ci]
…losure-backend-static into add-workflow-to-gen-data
This is an automated update by travis-ci at Wed May 29 21:25:38 PDT 2024 [skip ci]
…losure-backend-static into add-workflow-to-gen-data
ckingbailey
requested changes
Jun 3, 2024
| @@ -8,3 +8,11 @@ ENV PYTHONUNBUFFERED 1 | |||
| RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \ | |||
| && apt-get -y install --no-install-recommends postgresql-client ruby-full | |||
|
|
|||
| RUN pwd | |||
There was a problem hiding this comment.
Why do this? I can't see why the docker build would need to pwd
| @@ -26,8 +26,10 @@ services: | |||
| network_mode: service:db | |||
|
|
|||
| db: | |||
| #image: postgres:16.0-bullseye | |||
There was a problem hiding this comment.
Remove commented-out lines
| @@ -4,7 +4,7 @@ | |||
| "service": "app", | |||
| "workspaceFolder": "/workspace", | |||
| "remoteUser": "vscode", | |||
| "postCreateCommand": "bash ./.devcontainer/post-create-command.sh", | |||
| //"postCreateCommand": "bash ./.devcontainer/post-create-command.sh", | |||
There was a problem hiding this comment.
Remove commented-out line
| PGPASSWORD: app_password | ||
| services: | ||
| postgres: | ||
| #image: postgres:9.6-bullseye |
There was a problem hiding this comment.
Remove commented-out line
| #image: postgres:9.6-bullseye | ||
| image: postgres:15.6-bullseye | ||
| env: | ||
| POSTGRES_USER: app_user |
There was a problem hiding this comment.
Would be nice to have these in top-level env vars in the workflow so they can be shared between the two places they get used.
| name: "Generate Website Data" | ||
| on: | ||
| workflow_dispatch: | ||
| push: |
There was a problem hiding this comment.
Do we really want this to run on every single push? Do we have unlimited GH Actions hours?
| if: steps.filter.outputs.devcontainer == 'true' | ||
| run: | | ||
| docker build --no-cache --tag ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest -f ./.devcontainer/Dockerfile . | ||
| docker push ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest |
There was a problem hiding this comment.
Are tags immutable on GHCR? If not, why not use github.ref_name as the tag instead of a separate "sub-repo" for every commit?
Does GitHub let us fill their Docker registry with as many repos and tags as we care to? There's no limit? I'm surprised they'd be willing to host an image for every single commit, free of charge. Do the Docker images get pruned after some time or something?
| @@ -0,0 +1,15 @@ | |||
| CREATE TABLE committees_v2 ( | |||
There was a problem hiding this comment.
I appreciate that we're proactively defining these tables now instead of relying on csvsql to automagically do it for us.
ckingbailey
requested changes
Jun 4, 2024
| @@ -4,15 +4,21 @@ on: | |||
| jobs: | |||
| check: | |||
| runs-on: ubuntu-latest | |||
| container: | |||
| image: ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest | |||
There was a problem hiding this comment.
What happens here if no image has been pushed for github.ref_name?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The purpose of this pull request is to add a GitHub workflow for the ETL that runs in Travis CI. This workflow will be used to help integrate the download logic for netfile v2 into
make download.The csvkit that was being used was a beta version (0.9.0) and it was making it difficult to upgrade the ETL logic. I updated csvkit to the latest 1.3.0 version in another branch (PR) to address upgrade issues and merged it to this branch. The change includes making use of SQL files for creating the tables. These SQL files originated from csvsql 0.9.0. These files make the schema independent of the version of csvsql used and can be modified going forward as needed to change the schema of the tables.