Payments: intermediate refunds tables #3036

charlie-costanzo · 2023-10-19T19:17:56Z

Description

As a result of the payments refunds investigation this PR seeks to create two intermediate refunds tables and lightly modify the staging table:

int_payments__refunds_deduped
- table which unions the refunds observed as type = 'CREDIT' in the st_littlepay__micropayments table and the refunds from the st_littlepay__refunds table.
- Reconciles a common schema between the two
int_payments__refunds_to_aggregations
- table which summarizes refund amount activity on aggregation_id and retrieval_reference_number at a top level as well as by approval_status
stg_littlepay__refunds
- Adds some casting and trimming as necessary to facilitate downstream models
- Adds relationship tests with int_payments__refunds_deduped on aggregation_id and retrieval_reference_number
dbt documentation

Resolves #2978

Type of change

New feature
Documentation

How has this been tested?

dbt run locally and dbt test as necessary

Post-merge follow-ups

No action required

lauriemerrell

Pre-reviewing before an offline check-in about this tomorrow -- I think this generally looks good! I actually think that maybe we do want the refund amount to be negative? In https://github.com/cal-itp/data-infra/blob/main/warehouse/models/intermediate/payments/int_payments__settlements_summarized.sql I made credited (refunded) dollar amounts negative

lauriemerrell · 2023-10-19T22:55:19Z

warehouse/models/intermediate/payments/int_payments__refunds.sql

+        ABS(charge_amount) AS refund_amount,
+        EXTRACT(DATE FROM transaction_time AT TIME ZONE "America/Los_Angeles") AS transaction_date,
+        _line_number,
+        SAFE_CAST(currency_code AS NUMERIC) AS currency_code,


I actually wonder if this SAFE_CAST should happen in the staging table? Normally casting should happen in staging

github-actions · 2023-10-23T16:51:14Z

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.intermediate.payments.int_payments__refunds_deduped

calitp_warehouse.intermediate.payments.int_payments__refunds_to_aggregations

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

lauriemerrell

Overall looks good, just a few things, also I think you may need to rebase again -- diff is lookin' funky

lauriemerrell · 2023-10-23T17:59:42Z

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

+    FROM {{ ref('stg_littlepay__refunds') }}
+
+    -- should this be here, and should it also include 'AWAITING' if so?
+    WHERE approval_status != 'REFUSED'


I actually would not drop refused refunds because I think we want to see that a refund was requested even if it was refused

lauriemerrell · 2023-10-23T18:01:07Z

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

+    FROM refunds_union
+
+    -- for refunds that appear in both micropayments and refunds source tables,
+    -- is this a functional way to only keep duplicates that are in refunds source table?


I think that the number of refunds in this situation is like 3, so I would actually just manually drop them (where _key NOT IN ([list their keys])) from the micropayment refunds CTE above rather than doing this here

lauriemerrell · 2023-10-23T18:02:30Z

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

+
+    -- for refunds that appear in both micropayments and refunds source tables,
+    -- is this a functional way to only keep duplicates that are in refunds source table?
+    -- This is currently filtering out refunds table entries that have the same aggregation_id/refund_amount but different refund_ids


I actually think that we want the summarization to aggregations to be a separate table that uses a group by rather than a qualify; I think that it can be valid to have multiple refunds with the same aggregation ID so we want to group by aggregation ID and get the sum of the refund amounts (after deduping refunds here as you are above) rather than qualifying

Ok, after re-checking the example you provided offline I am less sure about this, it seems like some of these are legit just duplicates.

lauriemerrell · 2023-10-23T18:11:14Z

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

+        EXTRACT(DATE FROM transaction_time AT TIME ZONE "America/Los_Angeles") AS transaction_date,
+
+        -- Add columns that we want to preserve from refunds table after union as null strings
+        -- Is this an appropriate way to add new columns that are always NULL as the


lauriemerrell

Ok I think the SQL broadly looks good, biggest things are just to comment explanations of the various deduping steps and I think we need coalesced ID in the final qualify.

After that I think it's just documentation & testing.

For testing, I think that we want a test on stg_littlepay__refunds to test that there are no RRNs present in that table that aren't present here, and same for aggregation IDs. We want to make sure this table is capturing all the refunds.

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

lauriemerrell · 2023-10-25T19:17:58Z

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

+        source_table
+
+    FROM refunds_union
+    QUALIFY ROW_NUMBER() OVER (PARTITION BY retrieval_reference_number, refund_amount ORDER BY littlepay_export_ts DESC) = 1


I think this should be partitioned by coalesced_id because otherwise it will be dropping most copies of the rows missing RRN

…efund_id column to micropayments

…icropayment dup refunds

…d joins, coalesce retrieval_reference_number to use aggregation_id if not present

…tions

lauriemerrell

Two code comments and then for the PR description I think that you list int_payments__refunds_deduped twice, second one should be int_payments__refunds_to_aggregations, and at the bottom you reference stg_littlepay__micropayments but I think that should be stg_littlepay__refunds

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

lauriemerrell · 2023-10-30T18:05:25Z

warehouse/models/intermediate/payments/int_payments__refunds_to_aggregations.sql

+        aggregation_id,
+        retrieval_reference_number,
+        approval_status,
+        SUM(refund_amount) AS refund_amount,


I think this needs to be proposed_amount (this is what I've been complaining about, their column naming is incredibly counterintuitive -- proposed_amount is The actual amount refunded to the customer. while refund_amount is The amount equalling transaction_amount - proposed_amount.)

ah jeeze okay then I need to change how I prepare the micropayments table for the union in int_payments__refunds_rdeduped, doing that now

Ope wow yeah sorry I missed it above!

…enaming, added commment describing deduplication

lauriemerrell

Just one final thing I think!

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql

lauriemerrell

LGTM! 🎉

@charlie-costanzo

* Fix failing test on fct monthly routes (#2892) * refactor to address failing test and some docs * fix docs to reflect updated join * remove unused column * dbt modeling guidance docs (#2874) * start working on modeling decisions flowchart * fix admonition * fix single quotes * expand on bug identification and test linking * fixing bugs, grain overview * fix box type and more docs * example testing * add testing, documentation docs * fix links * rephrase some things, finish flowchat * fix emoji breaking mermaid and add callout about incremental models * add example stakeholder doc and rearrange a bit * move incremental warning, a few more tweaks * actually move incremental warning * pr comments * add loom links for debugging a failing test * reorder * tweak video link name * clarify bug types * more clarifications, esp for incremental models * linter * reference correct github action * dbt dev docs updates - linting and some extra context/links (#2896) * Support GTFS validator v4.1.0 in validation tasks (#2893) * Change GTFS RT / schedule mapping to handle schedule download failures (#2899) * change gtfs schedule mapping logic to better handle schedule download failures * update sql comment * bug fix: missing `mart_gtfs.fct_daily_scheduled_stops` (#2901) * fct_scheduled_stops uses fct_scheduled_trips as base, then left join * incremental where when trips had service, add test to check feed counts * remove incremental_where in dim_stop_times * 75% of trip feeds as threshold for check * Add dbt troubleshooting video (#2902) * add video of meeting to dbt warehouse docs page * add section on incremental models * add note to look at each section * typo add s to bugs * fix link ref * Bucket deprecation process update (#2898) * Clarify incremental docs (#2904) * clarify full refresh considerations * more tips on identifying incremental models * docs on the regular dbt dag task * move .github workflows readme * move k8s docs into readme * begin updating k8s docs * update cluster and jupyterhub docs * continue on k8s docs * add gitops diagram * fix marking mermaid for readme markdown not jupyterbook * remove period * couple tweaks * address PR feedback * style diagram * add note * clarify * rename build-dbt github action * fix typo * add docs about testing pod operators locally * add docs about schedule downloader secrets * Bump certifi from 2023.5.7 to 2023.7.22 in /services/gtfs-rt-archiver-v3 Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /ci Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /script Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-rt-parser-v2 Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-schedule-validator Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /warehouse Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /jobs/gtfs-aggregator-scraper Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2023.5.7 to 2023.7.22 in /apps/maps Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.5.7 to 2023.7.22. - [Commits](certifi/python-certifi@2023.05.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump tornado from 6.3.2 to 6.3.3 in /services/gtfs-rt-archiver-v3 Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.3.2...v6.3.3) --- updated-dependencies: - dependency-name: tornado dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi in /packages/calitp-data-analysis Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump certifi from 2022.12.7 to 2023.7.22 in /packages/calitp-data-infra Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * renaming missing form_factor values to 'Unidentified' (#2922) * Add mdformat to pre-commit (#2923) * include rides in (#2903) * revise payments date spine to exclude null values (#2924) * rearrange runbooks (#2926) * update k8s auth docs * Add Anaheim Transportation Network to LittlePay pipeline (#2928) * change fct_payments_rides_v2 join to include missing values (#2934) * make payments dags run hourly (#2933) * make payments dags run hourly * filter files to parse down to the hour, remove old daily outcomes * run dbt a bit earlier --------- Co-authored-by: Andrew Vaccaro <atvaccaro@gmail.com> * GTFS Validator Upgrade Follow-Up (#2935) * Update accepted ranges for validator version test * Update values list in _mart_gtfs_quality YAML * Switch bucket ref for external table to hourly bucket * Add seed for validator v4.1.0 rule details * add stub docs on which software to request for Git/VS Code on Caltrans PCs (#2938) * add stub docs on which software to request * format * add incomplete variable fares to device transaction types so they will appear in payments rides (#2940) * remove @charlie-costanzo from codeowners (#2939) Co-authored-by: Evan Siroky <evan.siroky@yahoo.com> * add popular packages * updated version number * imported packages except mypy * trying something i found on stackoverflow * rm init update func * rm optional and add return for geo utils * added return val and list[Any] using typing * geo utils:corr return type, changed back to list * geo_utils pivot_table optional str and cast * geo utils make pivot table optional[str] and literal * geo_utils changed to sequence * geo utils changed back to optional[str] * added # type ignore * geo_utils: replaced seq * geo_utils rm aggregate func * Windows git instructions (#2946) * finish windows git instructions * format * updated calitp-data-analysis package date * update dask req for altair * changed altair version * updated geopandas version,ran poetry add calitp * updated shared_utils and switch instead of checkout * double checked all checkout and shared_utils were changed * fixed typo * added trim_make_empty_string_null macro to littlepay staging tables (#2953) * rm colons, added pygis * (docs): clearer code blocks, update analytics info * (docs): clean up code blocks * move pygis link * Initial authorisations modeling (#2954) * clean strings and use lp source macro * use column macros to extract lp filename attributes * update macro file name and start duplicate handling * fix date macro * add tests and key construction * dedupe full dup rows and only do date imputation once * refactor macros for qualify dedupe statements * add descriptions in yaml * fix duplicate yaml anchor * relax uniqueness and drop simple dup rows * Payments: Remove over-aggressive file deduping (#2990) * remove over-aggressive file deduping * add content hash docs and yaml anchor for fuzzy uniqueness * remove references to me in non-payments locations (#2941) Co-authored-by: Evan Siroky <evan.siroky@yahoo.com> * add missing columns for anaheim and all agencies (#2992) * Update CODEOWNERS (#2991) * Update CODEOWNERS * precommit * Bump urllib3 from 1.26.16 to 1.26.17 in /ci (#2972) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.16 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.16...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 1.26.16 to 1.26.17 in /jobs/gtfs-aggregator-scraper (#2968) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.16 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.16...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Payments staging models refactor: settlements, micropayments, refunds (#2994) * fix deduping of same-timestamp rows * rename cte * handle micropayments in line with new approach * dedupe settlements * dedupe refunds * Payments: Add string handling, refactor modeling of Elavon data (#2957) * refactored staging table to focus on cleaning * broke out intermediate tables to focus on billing and deposits exclusively, and deduplicate * removed cleaning steps and refactored to focus on union of billing and deposits * replace previous transactions table * broke out intermediate tables to focus on billing and deposits exclusively, and deduplicate * made parse_elavon_date macro, implemented in stg_elavon_transactions * combined import CTEs for conciseness * implement qualify for intermediate table deduplication * changed elavon fact table name back to fct_elavon__transactions for the time being * removed _deduped from elavon intermediate tables since this is now handled in staging * fixed table names downstream for renamed tables * missed some renaming downstream * removed always null columns from int_elavon__billing_transactions * moved elavon deduplication to staging table * updated dbt docs * fix typo in stg_transit_database__funding_programs (#2985) * re-add columns in billing intermediate table to allow for union downstream, update dbt docs (#3002) * remove shape_array_key from yml (#3006) * Transit funding schema (#3003) * define schema explicitly * add coalesce * chore: update Codeowners (#3007) Include @tiffanychu90 as a codeowner of the warehouse. * fix: remove caltrans district from transit data quality issues table (#3005) This column is not included in the SQL and should be looked up via a bridge table anyways * Fix data type in transit funding programs schema (#3009) * remove dt from yml * fix data types * fix data types * Add agency information to fct_payments_rides_v2 (#3008) * updated docs for fct_payments_rides_v2 to include agency column descriptions * updated table fct_payments_rides_v2 to include agency columns agency_id and agency_name * Littlepay Table Deduplication Adjustments (#2993) * suppress duplicate row that is causing settlements test to fail (#3011) * Missing device transactions column (#3013) * add missing geography column back to device transactions * add geography field to yaml * fix join on feed_key (#3014) * Add new Littlepay columns + correct mistyped columns (#2996) * Add missing Littlepay columns to external tables * Correct column names and add new columns to models * Payments: Summarized authorisations model (#3001) * create authorisation deduping model and add upstream test * pivoted authorisations working wip * refactor payments row access policy macro for reusability * wip: summarize authorisations -- get latest rather than pivot * fix deduplication logic * completed summarized authorisations * remove unused yaml anchor * Payments: Intermediate settlements model (#3016) * stub: settlements intermediate model * wip settlements summarization: impute types and begin summarizing * working, finalized(?) settlements intermediate model * create docs macro for participant id column * summarizing settlements -- tweak refunds and dup handling * yaml updates * clarify descriptions of credits vs. debits * add column name * Suppress bad customer funding source row (#3021) * Expand Littlepay Sync Setup Docs (#3019) * Payments: update payments_gtfs_datasets seed to include elavon customer_name (#3024) * update payments_gtfs_datasets seed to include elavon customer name * update downstream models to use new seed column name * added all elavon customers to seed file, not just those that map to littlepay payments * removed not null test from littlepay_participant_id in payments seed (#3035) * Drop a few dupe micropayments (#3041) * Payments: Create `fct_payments_aggregations` model (#3040) * aggregate micropayments to aggregations * rename intermediate models * rename authorisations again to be even clearer * aggregations model * docs & yaml for aggregations model * add docs note per pr review * Payments: Add organization columns to mart tables for Metabase use (#3042) * add organization mapping to payments seed * rename payments mapping seed file * add organization information to payments mart; also add summarized aggregation date * Payments: Add end of month date columns (#3043) * add end of month dates * add not null test * Fix payments aggregation column names in YAML (#3051) * fix column name * fix other column name * Address a bunch of linter failures (#3054) * address a bunch of linter failures * more linter failures and fix some from before * more linter failures * one last one * Force python version for lint CI run --------- Co-authored-by: Soren Spicknall <soren@spicknall.us> * Create `fct_payments_settlements` (#3053) * payments fct settlements mart model * refactor intermediate model to use fct table * docs and yaml updates for fct payments settlements --------- Co-authored-by: Soren Spicknall <soren.s@jarv.us> * Take most recent authorisation with non-null status for aggregation (#3052) * Payments: intermediate refunds tables (#3036) * Payments: intermediate refunds table * make micropayments refunds that appear in charge_amount positive * add table config to file * moved some casting and cleaning to micropayments staging table * renamed table to include deduped, deduped refunds table, added null refund_id column to micropayments * added more deduplication logic to refunds intermediate table * changes from review * revised dedup of aggregation_id and refund_amount, manually exclude micropayment dup refunds * fixed IN clause * substitute retrieval_reference_number for aggregation_id in dedups and joins, coalesce retrieval_reference_number to use aggregation_id if not present * fix coalesce * revisions to coalesced_id, other fixes from review * added documentaion, relationship tests * attempting to get relationship test to work / persist all refunds * continued investigating failing relationship test on aggregation_id * successfully deduped with passing relationship test * create refunds to aggregations table int_payments__refunds_to_aggregations * added comment * add dbt docs for int_payments__refunds_to_aggregations * revise date extraction to exclude time zone * transitioned to using proposed_amount for sums, revised imports and renaming, added commment describing deduplication * Break up Elavon transactions by type & other small tweaks (#3058) * remove erroneous currency code column * change date of payments activity to always be payments date * working billing transactions mart table * create yaml and change column name * refactor to just use littlepay participant id directly * remove acct, routing columns * create deposits table * chargebacks model * adjustment transactions * use participant id directly for all transactions * add organization name to fct payments settlements * Fix two problematic Payments model tests (#3060) * change settlements test to point to mart table * add exception to test * Expand disk space usage and pod offset guidance (#3069) * Fare systems: Update columns & types (#3067) * change column type to JSON to handle change from string to array * fix columns and types for fare systems * Add aggregation ID to fct payments rides (#3077) * add aggregation id to payments rides * add test for aggregations appearing in fct aggregations * Add note about pygraphviz on ARM Macs (#3072) * Remove Principal Customer ID Tests W/Poor Assumptions (#3070) * Bump pillow from 9.5.0 to 10.0.1 in /warehouse (#2988) Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.5.0 to 10.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@9.5.0...10.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Soren Spicknall <soren.s@jarv.us> * Bump pillow from 9.5.0 to 10.0.1 in /jobs/gtfs-rt-parser-v2 (#2987) Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.5.0 to 10.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@9.5.0...10.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Soren Spicknall <soren.s@jarv.us> * Bump pillow from 10.0.0 to 10.0.1 in /images/jupyter-singleuser (#2986) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.0 to 10.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.0...10.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Soren Spicknall <soren.s@jarv.us> * Bump urllib3 from 1.26.17 to 1.26.18 in /jobs/gtfs-aggregator-scraper (#3034) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Soren Spicknall <soren.s@jarv.us> * Bump urllib3 v1 to 1.26.18 (#3083) * Update pillow and urllib3 in requirements.txt files (#3084) * Fix missing Littlepay transaction ids (#3075) * fix missing transaction ids/duplicate rides for historical data * adjust tests * HOTFIX: Pin version of Netlify CLI used in docs publication (#3087) * Payments documentation: Adding new agencies to Metabase (#3037) * Payments documentation: Adding new agencies to Metabase * fix typos, continue building out create new dashboard section * documentation updates, still needs editing * switched some headers for ordered lists * edits pt 1 * edits to move from draft PR to published * quick formatting fix * revisions based on PR review * revisions based on Soren's PR review * Restore Unpinned netlify-cli Version In Deps and Dockerfiles (#3089) * Improve warehouse README with tips/warnings (#3091) * Use More Recent nodejs Version In Build Tasks (#3090) * HOTFIX gpg command in Dockerfiles (#3094) * Resolve npm command non-recognition on warehouse build (#3099) * Expand JSON column values in the Benefits table into their own columns (#3081) Co-authored-by: Machiko Yasuda <machiko@compiler.la> Co-authored-by: Kegan Maher <kegan@compiler.la> * Improve warehouse README (part 2) (#3096) * Payments: Make customer processing participant-specific (#3109) * make customer processing participant-specific * update documentation * grammar * DAG: copy_prod_archiver_configs_to_test (#3110) * DAG: copy_prod_archiver_configs_to_test * reword dag description in metadata * update readme * changes based on PR review * got dag to work in local airflow * Create new table for Mobility Marketplace's new provider map (#3066) * Add diagram showing RBAC/security for Payments dashboards (#3125) * add diagram showing rbac/security for payments dashboards * add quotes per mermaid-js/mermaid#4388 (comment) * clarify labels * update TOC per PR review * Fix various outdated links in READMEs (#3130) * Bump urllib3 from 1.26.17 to 1.26.18 in /ci (#3033) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gitpython from 3.1.32 to 3.1.37 in /ci (#2999) Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.32 to 3.1.37. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.32...3.1.37) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gitpython from 3.1.32 to 3.1.37 in /images/jupyter-singleuser (#3000) Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.32 to 3.1.37. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.32...3.1.37) --- updated-dependencies: - dependency-name: gitpython dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump aiohttp to 3.8.6 (#3131) * Update minor deps that require heightened access for CI runs (#3135) * Upgrade pyarrow dep in Jupyterhub image and data analysis image (#3136) * Unpin Python version for linting CI task (#3137) * add line number to sorting to make it deterministic (#3138) * add refund id (#3151) * Linkfix in /airflow README (#3154) * Settlements schema updates (#3153) * add new columns to settlements staging table * rename column * add yaml for fct settlements * settlements to aggregations updates * new fields on fct aggregations * Refactor payments rides for code readability and modularity: micropayments portion (#3123) * updates to refunds deduped and its documentation * working micropayments intermediate model * wip: docs macros for micropayments * remove unused docstring * keep migrating lp docs to docs macros * keep updating docs for micropayments model * more yaml documentation * update refunds modeling to accommodate micropayments modeling * finish adding yaml * wip: further dedupe refunds where refund id changes * make refund logic more sophisticated to handle id mmismatches * fix issue from rebase * use intermediate model for micropayments transformations * docs updates * Infra Docs Overhaul Part 1 - READMEs and Lagging Versions (#3156) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Laurie <55149902+lauriemerrell@users.noreply.github.com> Co-authored-by: Andrew Vaccaro <atvaccaro@users.noreply.github.com> Co-authored-by: tiffanychu90 <49657200+tiffanychu90@users.noreply.github.com> Co-authored-by: Andrew Vaccaro <atvaccaro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Charlie Costanzo <cvc5185@gmail.com> Co-authored-by: Eric Dasmalchi <54728455+edasmalchi@users.noreply.github.com> Co-authored-by: Evan Siroky <evan.siroky@yahoo.com> Co-authored-by: amandaha8 <amandahastuyck@gmail.com> Co-authored-by: Amanda <95305899+amandaha8@users.noreply.github.com> Co-authored-by: tiffanychu90 <tiffany.chu@dot.ca.gov> Co-authored-by: Vladimir Jimenez <allejo@me.com> Co-authored-by: Vladimir Jimenez <vlad@compiler.la> Co-authored-by: Machiko Yasuda <machiko@compiler.la> Co-authored-by: Kegan Maher <kegan@compiler.la> Co-authored-by: Github Action build-release-candidate <runner@fv-az1269-152>

charlie-costanzo self-assigned this Oct 19, 2023

lauriemerrell reviewed Oct 19, 2023

View reviewed changes

charlie-costanzo requested a review from lauriemerrell October 23, 2023 17:45

lauriemerrell reviewed Oct 23, 2023

View reviewed changes

charlie-costanzo force-pushed the payments-int-refunds-table branch 2 times, most recently from bb41840 to b84aae9 Compare October 24, 2023 16:49

lauriemerrell reviewed Oct 25, 2023

View reviewed changes

charlie-costanzo requested a review from lauriemerrell October 26, 2023 14:14

charlie-costanzo force-pushed the payments-int-refunds-table branch 2 times, most recently from 40f2893 to 8f08976 Compare October 26, 2023 22:09

charlie-costanzo added 15 commits October 30, 2023 11:28

Payments: intermediate refunds table

1a861e1

make micropayments refunds that appear in charge_amount positive

a07a3a5

add table config to file

eb2c935

moved some casting and cleaning to micropayments staging table

6f8e2a7

renamed table to include deduped, deduped refunds table, added null r…

0785a15

…efund_id column to micropayments

added more deduplication logic to refunds intermediate table

7d6392b

changes from review

8f1ef8a

revised dedup of aggregation_id and refund_amount, manually exclude m…

0e1254a

…icropayment dup refunds

fixed IN clause

88181ca

substitute retrieval_reference_number for aggregation_id in dedups an…

f35ee0e

…d joins, coalesce retrieval_reference_number to use aggregation_id if not present

fix coalesce

32e7f12

revisions to coalesced_id, other fixes from review

9b35fce

added documentaion, relationship tests

d1b1698

attempting to get relationship test to work / persist all refunds

a6d5e00

continued investigating failing relationship test on aggregation_id

c847c72

charlie-costanzo force-pushed the payments-int-refunds-table branch from 8f08976 to c847c72 Compare October 30, 2023 15:28

charlie-costanzo added 3 commits October 30, 2023 11:39

successfully deduped with passing relationship test

b0b36b9

create refunds to aggregations table int_payments__refunds_to_aggrega…

a01f4e5

…tions

added comment

9adf539

add dbt docs for int_payments__refunds_to_aggregations

7609f83

charlie-costanzo changed the title ~~Payments: intermediate refunds table~~ Payments: intermediate refunds tables Oct 30, 2023

revise date extraction to exclude time zone

8050eba

charlie-costanzo marked this pull request as ready for review October 30, 2023 17:47

charlie-costanzo requested review from evansiroky, SorenSpicknall and tiffanychu90 as code owners October 30, 2023 17:47

lauriemerrell reviewed Oct 30, 2023

View reviewed changes

transitioned to using proposed_amount for sums, revised imports and r…

09c50fe

…enaming, added commment describing deduplication

charlie-costanzo requested a review from lauriemerrell October 30, 2023 18:34

lauriemerrell reviewed Oct 30, 2023

View reviewed changes

warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql Show resolved Hide resolved

lauriemerrell approved these changes Oct 30, 2023

View reviewed changes

charlie-costanzo merged commit b303906 into main Oct 30, 2023
4 checks passed

charlie-costanzo deleted the payments-int-refunds-table branch October 30, 2023 18:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Payments: intermediate refunds tables #3036

Payments: intermediate refunds tables #3036

charlie-costanzo commented Oct 19, 2023 •

edited

lauriemerrell left a comment

lauriemerrell Oct 19, 2023

github-actions bot commented Oct 23, 2023 •

edited

lauriemerrell left a comment

lauriemerrell Oct 23, 2023

lauriemerrell Oct 23, 2023

lauriemerrell Oct 23, 2023

lauriemerrell Oct 23, 2023

lauriemerrell Oct 23, 2023

lauriemerrell left a comment

lauriemerrell Oct 25, 2023

lauriemerrell left a comment

lauriemerrell Oct 30, 2023

charlie-costanzo Oct 30, 2023

lauriemerrell Oct 30, 2023

lauriemerrell left a comment

lauriemerrell left a comment

Payments: intermediate refunds tables #3036

Payments: intermediate refunds tables #3036

Conversation

charlie-costanzo commented Oct 19, 2023 • edited

Description

Type of change

How has this been tested?

Post-merge follow-ups

lauriemerrell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 23, 2023 • edited

Checks/potential follow-ups

New models 🌱

DAG

lauriemerrell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lauriemerrell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lauriemerrell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lauriemerrell left a comment

Choose a reason for hiding this comment

lauriemerrell left a comment

Choose a reason for hiding this comment

charlie-costanzo commented Oct 19, 2023 •

edited

github-actions bot commented Oct 23, 2023 •

edited