Skip to content

Pipelines automation #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 69 commits into from
Sep 9, 2024
Merged

Pipelines automation #3

merged 69 commits into from
Sep 9, 2024

Conversation

max-ostapenko
Copy link
Contributor

@max-ostapenko max-ostapenko commented Aug 27, 2024 •
edited
Loading

Automation with Workflows:

  • before crawl (polling BQ tables)
  • after crawl

For a list of supported pipelines and a corresponding triggers see README.

Please also read through notes in the pipeline doc.

Closes HTTPArchive/cwv-tech-report#36

TODO before prod ready:

  1. Dataform service account has only read-only access to BigQuery (to avoid messing with tables before tested). Needs to be updated with write access to particular datasets.
  2. FYI Dataform repo using PAT, so all the changes done via Dataform console will be committed on behalf of @max-ostapenko. Do we need any action?

max-ostapenko and others added 27 commits August 22, 2024 22:03
@max-ostapenko
pages and legacy lighthouse
e769d00
@max-ostapenko
fix
2933e2a
@max-ostapenko
first test
d7da0be
@max-ostapenko
pages reference
bb79780
@max-ostapenko
pages insert
d11853f
@max-ostapenko
date var
326a062
@max-ostapenko
js rewrite
0ca431f
@max-ostapenko
dataset
e325e74
@max-ostapenko
Initial commit
7f967d7
@max-ostapenko
init
8416aa1
@max-ostapenko
migrated to external repo
f7a217d
@max-ostapenko
core_web_vitals
f2bd92d
@max-ostapenko
clean graph, tested
1e81298
@max-ostapenko
publish core_web_vitals.technologies
e7ab825
@max-ostapenko
Dev (#1)
ee4119c 
* workspace init

* pages and legacy lighthouse

* fix

* first test

* pages insert

* date var

* js rewrite

* dataset

* core_web_vitals

* clean graph, tested

* publish core_web_vitals.technologies
@max-ostapenko
technologies partitioning
59e6bbf
@max-ostapenko
sync
a54bc56
@max-ostapenko
past month date for cwv
13ea93b
@max-ostapenko
8pm
01ae1c5
@max-ostapenko
package-lock.json
1b67347
@max-ostapenko
ignore full-refresh
b48d1c1
@max-ostapenko
readme
d4e0ef7
@max-ostapenko
updated tags and example assert
90374ae
@max-ostapenko
dependency assertions
66cb2dc
@max-ostapenko
current month commented
ecbbc40
@max-ostapenko
assert fix
ce2d190
@max-ostapenko
all tables publish
f903e39
@max-ostapenko max-ostapenko mentioned this pull request Aug 27, 2024
Merged
14 tasks
@max-ostapenko max-ostapenko changed the title all tables publish Finalize the pipelines automation v2 Aug 27, 2024
max-ostapenko
max-ostapenko commented Sep 1, 2024
View reviewed changes
max-ostapenko
max-ostapenko commented Sep 1, 2024
View reviewed changes
max-ostapenko
max-ostapenko commented Sep 1, 2024
View reviewed changes
@max-ostapenko max-ostapenko mentioned this pull request Sep 3, 2024
Merged
4 tasks
@pmeenan pmeenan removed their request for review September 3, 2024 20:12
@pmeenan
Copy link
Member

pmeenan commented Sep 3, 2024

Just pulling myself off the reviewers list since I'm not all that familiar with the back-end data pipelines (other than writing data to them).

@max-ostapenko
Copy link
Contributor Author

The pipeline automation PR is complete.
The crawl starts next Tuesday (so should the CWV tech report pipeline).

@tunetheweb Can I help so we get this reviewed by then and see the first pipeline results?

tunetheweb
tunetheweb reviewed Sep 8, 2024
View reviewed changes
@max-ostapenko @tunetheweb
Update definitions/output/all/parsed_css.js
a6ac56e 
Co-authored-by: Barry Pollard <barrypollard@google.com>
tunetheweb
tunetheweb approved these changes Sep 9, 2024
View reviewed changes
Copy link
Member

@tunetheweb tunetheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Really excited about automating this.

@max-ostapenko max-ostapenko merged commit a588217 into main Sep 9, 2024
@max-ostapenko max-ostapenko deleted the dev branch September 9, 2024 18:51
This was referenced Sep 9, 2024
Closed
Closed
@max-ostapenko max-ostapenko mentioned this pull request Sep 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tunetheweb tunetheweb tunetheweb approved these changes

@rviscomi rviscomi Awaiting requested review from rviscomi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Automate populating tech report SQL
3 participants
@max-ostapenko @tunetheweb @pmeenan