Skip to content

Commit

Permalink
Invoke Prod Release (#3158)
Browse files Browse the repository at this point in the history
* Fix failing test on fct monthly routes (#2892)

* refactor to address failing test and some docs

* fix docs to reflect updated join

* remove unused column

* dbt modeling guidance docs (#2874)

* start working on modeling decisions flowchart

* fix admonition

* fix single quotes

* expand on bug identification and test linking

* fixing bugs, grain overview

* fix box type and more docs

* example testing

* add testing, documentation docs

* fix links

* rephrase some things, finish flowchat

* fix emoji breaking mermaid and add callout about incremental models

* add example stakeholder doc and rearrange a bit

* move incremental warning, a few more tweaks

* actually move incremental warning

* pr comments

* add loom links for debugging a failing test

* reorder

* tweak video link name

* clarify bug types

* more clarifications, esp for incremental models

* linter

* reference correct github action

* dbt dev docs updates - linting and some extra context/links (#2896)

* Support GTFS validator v4.1.0 in validation tasks (#2893)

* Change GTFS RT / schedule mapping to handle schedule download failures (#2899)

* change gtfs schedule mapping logic to better handle schedule download failures

* update sql comment

* bug fix: missing `mart_gtfs.fct_daily_scheduled_stops` (#2901)

* fct_scheduled_stops uses fct_scheduled_trips as base, then left join

* incremental where when trips had service, add test to check feed counts

* remove incremental_where in dim_stop_times

* 75% of trip feeds as threshold for check

* Add dbt troubleshooting video (#2902)

* add video of meeting to dbt warehouse docs page

* add section on incremental models

* add note to look at each section

* typo add s to bugs

* fix link ref

* Bucket deprecation process update (#2898)

* Clarify incremental docs (#2904)

* clarify full refresh considerations

* more tips on identifying incremental models

* docs on the regular dbt dag task

* move .github workflows readme

* move k8s docs into readme

* begin updating k8s docs

* update cluster and jupyterhub docs

* continue on k8s docs

* add gitops diagram

* fix marking mermaid for readme markdown not jupyterbook

* remove period

* couple tweaks

* address PR feedback

* style diagram

* add note

* clarify

* rename build-dbt github action

* fix typo

* add docs about testing pod operators locally

* add docs about schedule downloader secrets

* Bump certifi from 2023.5.7 to 2023.7.22 in /services/gtfs-rt-archiver-v3

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /ci

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /script

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-rt-parser-v2

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-schedule-validator

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /warehouse

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-aggregator-scraper

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2023.5.7 to 2023.7.22 in /apps/maps

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2023.05.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump tornado from 6.3.2 to 6.3.3 in /services/gtfs-rt-archiver-v3

Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi in /packages/calitp-data-analysis

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump certifi from 2022.12.7 to 2023.7.22 in /packages/calitp-data-infra

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* renaming missing form_factor values to 'Unidentified' (#2922)

* Add mdformat to pre-commit (#2923)

* include  rides in (#2903)

* revise payments date spine to exclude null values (#2924)

* rearrange runbooks (#2926)

* update k8s auth docs

* Add Anaheim Transportation Network to LittlePay pipeline (#2928)

* change fct_payments_rides_v2 join to include missing values (#2934)

* make payments dags run hourly (#2933)

* make payments dags run hourly

* filter files to parse down to the hour, remove old daily outcomes

* run dbt a bit earlier

---------

Co-authored-by: Andrew Vaccaro <atvaccaro@gmail.com>

* GTFS Validator Upgrade Follow-Up (#2935)

* Update accepted ranges for validator version test

* Update values list in _mart_gtfs_quality YAML

* Switch bucket ref for external table to hourly bucket

* Add seed for validator v4.1.0 rule details

* add stub docs on which software to request for Git/VS Code on Caltrans PCs (#2938)

* add stub docs on which software to request

* format

* add incomplete variable fares to device transaction types so they will appear in payments rides (#2940)

* remove @charlie-costanzo from codeowners (#2939)

Co-authored-by: Evan Siroky <evan.siroky@yahoo.com>

* add popular packages

* updated version number

* imported packages except mypy

* trying something i found on stackoverflow

* rm init update func

* rm optional and add return for geo utils

* added return val and list[Any] using typing

* geo utils:corr return type, changed back to list

* geo_utils pivot_table optional str and cast

* geo utils make pivot table optional[str] and literal

* geo_utils changed to sequence

* geo utils changed back to optional[str]

* added # type ignore

* geo_utils: replaced seq

* geo_utils rm aggregate func

* Windows git instructions (#2946)

* finish windows git instructions

* format

* updated calitp-data-analysis package date

* update dask req for altair

* changed altair version

* updated geopandas version,ran poetry add calitp

* updated shared_utils and switch instead of checkout

* double checked all checkout and shared_utils were changed

* fixed typo

* added trim_make_empty_string_null macro to littlepay staging tables (#2953)

* rm colons, added pygis

* (docs): clearer code blocks, update analytics info

* (docs): clean up code blocks

* move pygis link

* Initial authorisations modeling (#2954)

* clean strings and use lp source macro

* use column macros to extract lp filename attributes

* update macro file name and start duplicate handling

* fix date macro

* add tests and key construction

* dedupe full dup rows and only do date imputation once

* refactor macros for qualify dedupe statements

* add descriptions in yaml

* fix duplicate yaml anchor

* relax uniqueness and drop simple dup rows

* Payments: Remove over-aggressive file deduping (#2990)

* remove over-aggressive file deduping

* add content hash docs and yaml anchor for fuzzy uniqueness

* remove references to me in non-payments locations (#2941)

Co-authored-by: Evan Siroky <evan.siroky@yahoo.com>

* add missing columns for anaheim and all agencies (#2992)

* Update CODEOWNERS (#2991)

* Update CODEOWNERS

* precommit

* Bump urllib3 from 1.26.16 to 1.26.17 in /ci (#2972)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.16 to 1.26.17.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.16...1.26.17)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump urllib3 from 1.26.16 to 1.26.17 in /jobs/gtfs-aggregator-scraper (#2968)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.16 to 1.26.17.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.16...1.26.17)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Payments staging models refactor: settlements, micropayments, refunds (#2994)

* fix deduping of same-timestamp rows

* rename cte

* handle micropayments in line with new approach

* dedupe settlements

* dedupe refunds

* Payments: Add string handling, refactor modeling of Elavon data (#2957)

* refactored staging table to focus on cleaning

* broke out intermediate tables to focus on billing and deposits exclusively, and deduplicate

* removed cleaning steps and refactored to focus on union of billing and deposits

* replace previous transactions table

* broke out intermediate tables to focus on billing and deposits exclusively, and deduplicate

* made parse_elavon_date macro, implemented in stg_elavon_transactions

* combined import CTEs for conciseness

* implement qualify for intermediate table deduplication

* changed elavon fact table name back to fct_elavon__transactions for the time being

* removed _deduped from elavon intermediate tables since this is now handled in staging

* fixed table names downstream for renamed tables

* missed some renaming downstream

* removed always null columns from int_elavon__billing_transactions

* moved elavon deduplication to staging table

* updated dbt docs

* fix typo in stg_transit_database__funding_programs (#2985)

* re-add columns in billing intermediate table to allow for union downstream, update dbt docs (#3002)

* remove shape_array_key from yml (#3006)

* Transit funding schema (#3003)

* define schema explicitly

* add coalesce

* chore: update Codeowners (#3007)

Include @tiffanychu90 as a codeowner of the warehouse.

* fix: remove caltrans district from transit data quality issues table (#3005)

This column is not included in the SQL and should be looked up via a bridge table anyways

* Fix data type in transit funding programs schema (#3009)

* remove dt from yml

* fix data types

* fix data types

* Add agency information to fct_payments_rides_v2 (#3008)

* updated docs for fct_payments_rides_v2 to include agency column descriptions

* updated table fct_payments_rides_v2 to include agency columns agency_id and agency_name

* Littlepay Table Deduplication Adjustments (#2993)

* suppress duplicate row that is causing settlements test to fail (#3011)

* Missing device transactions column (#3013)

* add missing geography column back to device transactions

* add geography field to yaml

* fix join on feed_key (#3014)

* Add new Littlepay columns + correct mistyped columns (#2996)

* Add missing Littlepay columns to external tables

* Correct column names and add new columns to models

* Payments: Summarized authorisations model  (#3001)

* create authorisation deduping model and add upstream test

* pivoted authorisations working wip

* refactor payments row access policy macro for reusability

* wip: summarize authorisations -- get latest rather than pivot

* fix deduplication logic

* completed summarized authorisations

* remove unused yaml anchor

* Payments: Intermediate settlements model (#3016)

* stub: settlements intermediate model

* wip settlements summarization: impute types and begin summarizing

* working, finalized(?) settlements intermediate model

* create docs macro for participant id column

* summarizing settlements -- tweak refunds and dup handling

* yaml updates

* clarify descriptions of credits vs. debits

* add column name

* Suppress bad customer funding source row (#3021)

* Expand Littlepay Sync Setup Docs (#3019)

* Payments: update payments_gtfs_datasets seed to include elavon customer_name (#3024)

* update payments_gtfs_datasets seed to include elavon customer name

* update downstream models to use new seed column name

* added all elavon customers to seed file, not just those that map to littlepay payments

* removed not null test from littlepay_participant_id in payments seed (#3035)

* Drop a few dupe micropayments (#3041)

* Payments: Create `fct_payments_aggregations` model (#3040)

* aggregate micropayments to aggregations

* rename intermediate models

* rename authorisations again to be even clearer

* aggregations model

* docs & yaml for aggregations model

* add docs note per pr review

* Payments: Add organization columns to mart tables for Metabase use (#3042)

* add organization mapping to payments seed

* rename payments mapping seed file

* add organization information to payments mart; also add summarized aggregation date

* Payments: Add end of month date columns (#3043)

* add end of month dates

* add not null test

* Fix payments aggregation column names in YAML (#3051)

* fix column name

* fix other column name

* Address a bunch of linter failures (#3054)

* address a bunch of linter failures

* more linter failures and fix some from before

* more linter failures

* one last one

* Force python version for lint CI run

---------

Co-authored-by: Soren Spicknall <soren@spicknall.us>

* Create `fct_payments_settlements` (#3053)

* payments fct settlements mart model

* refactor intermediate model to use fct table

* docs and yaml updates for fct payments settlements

---------

Co-authored-by: Soren Spicknall <soren.s@jarv.us>

* Take most recent authorisation with non-null status for aggregation (#3052)

* Payments: intermediate refunds tables (#3036)

* Payments: intermediate refunds table

* make micropayments refunds that appear in charge_amount positive

* add table config to file

* moved some casting and cleaning to micropayments staging table

* renamed table to include deduped, deduped refunds table, added null refund_id column to micropayments

* added more deduplication logic to refunds intermediate table

* changes from review

* revised dedup of aggregation_id and refund_amount, manually exclude micropayment dup refunds

* fixed IN clause

* substitute retrieval_reference_number for aggregation_id in dedups and joins, coalesce retrieval_reference_number to use aggregation_id if not present

* fix coalesce

* revisions to coalesced_id, other fixes from review

* added documentaion, relationship tests

* attempting to get relationship test to work / persist all refunds

* continued investigating failing relationship test on aggregation_id

* successfully deduped with passing relationship test

* create refunds to aggregations table int_payments__refunds_to_aggregations

* added comment

* add dbt docs for int_payments__refunds_to_aggregations

* revise date extraction to exclude time zone

* transitioned to using proposed_amount for sums, revised imports and renaming, added commment describing deduplication

* Break up Elavon transactions by type & other small tweaks (#3058)

* remove erroneous currency code column

* change date of payments activity to always be payments date

* working billing transactions mart table

* create yaml and change column name

* refactor to just use littlepay participant id directly

* remove acct, routing columns

* create deposits table

* chargebacks model

* adjustment transactions

* use participant id directly for all transactions

* add organization name to fct payments settlements

* Fix two problematic Payments model tests (#3060)

* change settlements test to point to mart table

* add exception to test

* Expand disk space usage and pod offset guidance (#3069)

* Fare systems: Update columns & types (#3067)

* change column type to JSON to handle change from string to array

* fix columns and types for fare systems

* Add aggregation ID to fct payments rides (#3077)

* add aggregation id to payments rides

* add test for aggregations appearing in fct aggregations

* Add note about pygraphviz on ARM Macs (#3072)

* Remove Principal Customer ID Tests W/Poor Assumptions (#3070)

* Bump pillow from 9.5.0 to 10.0.1 in /warehouse (#2988)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.5.0 to 10.0.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@9.5.0...10.0.1)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Soren Spicknall <soren.s@jarv.us>

* Bump pillow from 9.5.0 to 10.0.1 in /jobs/gtfs-rt-parser-v2 (#2987)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.5.0 to 10.0.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@9.5.0...10.0.1)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Soren Spicknall <soren.s@jarv.us>

* Bump pillow from 10.0.0 to 10.0.1 in /images/jupyter-singleuser (#2986)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.0 to 10.0.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.0.0...10.0.1)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Soren Spicknall <soren.s@jarv.us>

* Bump urllib3 from 1.26.17 to 1.26.18 in /jobs/gtfs-aggregator-scraper (#3034)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.17...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Soren Spicknall <soren.s@jarv.us>

* Bump urllib3 v1 to 1.26.18 (#3083)

* Update pillow and urllib3 in requirements.txt files (#3084)

* Fix missing Littlepay transaction ids (#3075)

* fix missing transaction ids/duplicate rides for historical data

* adjust tests

* HOTFIX: Pin version of Netlify CLI used in docs publication (#3087)

* Payments documentation: Adding new agencies to Metabase (#3037)

* Payments documentation: Adding new agencies to Metabase

* fix typos, continue building out create new dashboard section

* documentation updates, still needs editing

* switched some headers for ordered lists

* edits pt 1

* edits to move from draft PR to published

* quick formatting fix

* revisions based on PR review

* revisions based on Soren's PR review

* Restore Unpinned netlify-cli Version In Deps and Dockerfiles (#3089)

* Improve warehouse README with tips/warnings (#3091)

* Use More Recent nodejs Version In Build Tasks (#3090)

* HOTFIX gpg command in Dockerfiles (#3094)

* Resolve npm command non-recognition on warehouse build (#3099)

* Expand JSON column values in the Benefits table into their own columns (#3081)

Co-authored-by: Machiko Yasuda <machiko@compiler.la>
Co-authored-by: Kegan Maher <kegan@compiler.la>

* Improve warehouse README (part 2) (#3096)

* Payments: Make customer processing participant-specific (#3109)

* make customer processing participant-specific

* update documentation

* grammar

* DAG: copy_prod_archiver_configs_to_test (#3110)

* DAG: copy_prod_archiver_configs_to_test

* reword dag description in metadata

* update readme

* changes based on PR review

* got dag to work in local airflow

* Create new table for Mobility Marketplace's new provider map (#3066)

* Add diagram showing RBAC/security for Payments dashboards (#3125)

* add diagram showing rbac/security for payments dashboards

* add quotes per mermaid-js/mermaid#4388 (comment)

* clarify labels

* update TOC per PR review

* Fix various outdated links in READMEs (#3130)

* Bump urllib3 from 1.26.17 to 1.26.18 in /ci (#3033)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.17...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump gitpython from 3.1.32 to 3.1.37 in /ci (#2999)

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.32 to 3.1.37.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](gitpython-developers/GitPython@3.1.32...3.1.37)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump gitpython from 3.1.32 to 3.1.37 in /images/jupyter-singleuser (#3000)

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.32 to 3.1.37.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](gitpython-developers/GitPython@3.1.32...3.1.37)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump aiohttp to 3.8.6 (#3131)

* Update minor deps that require heightened access for CI runs (#3135)

* Upgrade pyarrow dep in Jupyterhub image and data analysis image (#3136)

* Unpin Python version for linting CI task (#3137)

* add line number to sorting to make it deterministic (#3138)

* add refund id (#3151)

* Linkfix in /airflow README (#3154)

* Settlements schema updates (#3153)

* add new columns to settlements staging table

* rename column

* add yaml for fct settlements

* settlements to aggregations updates

* new fields on fct aggregations

* Refactor payments rides for code readability and modularity: micropayments portion (#3123)

* updates to refunds deduped and its documentation

* working micropayments intermediate model

* wip: docs macros for micropayments

* remove unused docstring

* keep migrating lp docs to docs macros

* keep updating docs for micropayments model

* more yaml documentation

* update refunds modeling to accommodate micropayments modeling

* finish adding yaml

* wip: further dedupe refunds where refund id changes

* make refund logic more sophisticated to handle id mmismatches

* fix issue from rebase

* use intermediate model for micropayments transformations

* docs updates

* Infra Docs Overhaul Part 1 - READMEs and Lagging Versions (#3156)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Laurie <55149902+lauriemerrell@users.noreply.github.com>
Co-authored-by: Andrew Vaccaro <atvaccaro@users.noreply.github.com>
Co-authored-by: tiffanychu90 <49657200+tiffanychu90@users.noreply.github.com>
Co-authored-by: Andrew Vaccaro <atvaccaro@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Charlie Costanzo <cvc5185@gmail.com>
Co-authored-by: Eric Dasmalchi <54728455+edasmalchi@users.noreply.github.com>
Co-authored-by: Evan Siroky <evan.siroky@yahoo.com>
Co-authored-by: amandaha8 <amandahastuyck@gmail.com>
Co-authored-by: Amanda <95305899+amandaha8@users.noreply.github.com>
Co-authored-by: tiffanychu90 <tiffany.chu@dot.ca.gov>
Co-authored-by: Vladimir Jimenez <allejo@me.com>
Co-authored-by: Vladimir Jimenez <vlad@compiler.la>
Co-authored-by: Machiko Yasuda <machiko@compiler.la>
Co-authored-by: Kegan Maher <kegan@compiler.la>
Co-authored-by: Github Action build-release-candidate <runner@fv-az1269-152>
  • Loading branch information
17 people committed Nov 30, 2023
1 parent 3c92d97 commit 78fd526
Show file tree
Hide file tree
Showing 5 changed files with 192 additions and 107 deletions.
27 changes: 27 additions & 0 deletions ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ a deployment named `archiver` is configured in [the prod channel](./channels/pro
by `invoke` (see below) calling `kubectl` commands.

## invoke (aka pyinvoke)

[invoke](https://docs.pyinvoke.org/en/stable/) is a Python framework for executing subprocesses and building a CLI application.
The tasks are defined in `tasks.py` and configuration in `invoke.yaml`; config values under the top-level `calitp`
are specific to our defined tasks.
Expand All @@ -15,3 +16,29 @@ Individual release channels/environments are config files that are passed to inv
```bash
poetry run invoke release -f channels/test.yaml
```

## GitOps

In this diagram, arrows represent human actions such as opening and merging PRs and nodes (except for the very first) represent automated actions such as `invoke` deploying to the cluster. Green nodes indicate a deployment while white nodes indicate an automated git action such as branch creation or commenting on a pull request.

```mermaid
flowchart TD
classDef default fill:white, color:black, stroke:black
classDef initial fill:lightblue, color:black
classDef deploy fill:lightgreen, color:black
pr[Push commits to a branch.\nDoes a test environment exist?]
candidates_branch[GitHub Action renders candidates/branch-name]
branch_diff[invoke diff renders on test PR]
branch_invoke[invoke releases to test]
candidates_main[GitHub Action builds images and renders candidates/main\nNote: if you stop here, no Kubernetes changes will actually be deployed.]
prod_diff[invoke diff renders on prod PR]
prod_invoke[invoke releases to prod]
pr -- Yes --> candidates_branch -- "Open PR from candidates/branch-name to releases/test" --> branch_diff -- "Merge candidate PR to releases/test" --> branch_invoke -- Merge original PR to main after review and testing --> candidates_main -- "Open PR from candidates/main to releases/prod" --> prod_diff -- "Merge candidate PR to releases/prod" --> prod_invoke
pr -- "No; merge to main after review" --> candidates_main
class pr initial
class branch_invoke,prod_invoke deploy
```

0 comments on commit 78fd526

Please sign in to comment.