Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename and test XBRL metadata calculations #2563

Merged
merged 114 commits into from
Jun 8, 2023
Merged
Show file tree
Hide file tree
Changes from 96 commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
3135fed
very wip-y xbrl calculation metadata rename process
cmgosnell May 3, 2023
f6e19f8
fix col rename for mutli-WideToTidy params
cmgosnell May 4, 2023
344bac7
clean docs bc lists are hard apparently (for me!)
cmgosnell May 4, 2023
2a82b46
per-table calc name checking
cmgosnell May 4, 2023
d7c3722
very rough pass at checking if the calcs calc for multiple tables
cmgosnell May 4, 2023
f7aeacc
lol remove the old table-transformer metadata translation
cmgosnell May 4, 2023
b547f4f
Fix income_statement_ferc1 utility_type categorization bug
zaneselvans May 4, 2023
08165a6
clean up the ferc1_transform_asset_factory for the glue test
cmgosnell May 5, 2023
4edb224
Merge branch 'dev' into xbrl_meta_reshape
cmgosnell May 5, 2023
068f223
Merge pull request #2565 from catalyst-cooperative/exploding-income-s…
cmgosnell May 5, 2023
0a90510
integrate manual updates to calcuations
cmgosnell May 5, 2023
d5242cd
insert the warning about missing xbrl_factoids
cmgosnell May 5, 2023
43a60ad
fix missing names in the electric_operating_expenses_ferc1 table calcs
cmgosnell May 5, 2023
20b09db
update the off calc by using np.isclose
cmgosnell May 5, 2023
e7fc08e
enable more manual calc fixes and clean electric_operating_expenses_f…
cmgosnell May 6, 2023
b91850c
make null additions/removals of xbrl cacls a lil more straightforward
cmgosnell May 8, 2023
a901680
add notebook for ferc1 table explosion exploration
cmgosnell May 8, 2023
a5cb358
rename the silly method name
cmgosnell May 8, 2023
6519570
Fix remaining calculation errors in income_statement_ferc1
zaneselvans May 8, 2023
9fde7b3
iadd source_table into the converted metadata
cmgosnell May 8, 2023
370a5c2
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell May 8, 2023
38f18ff
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 8, 2023
388be16
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 8, 2023
e039579
Add metadata cols to electric operating revenues table
zaneselvans May 8, 2023
d5e865c
enable (but not actually use yet) a multi-table explosion via source_…
cmgosnell May 9, 2023
27e2af6
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 10, 2023
5f0fa48
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 11, 2023
dd0931c
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
zaneselvans May 11, 2023
c5b6a8d
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 15, 2023
febad7a
Merge branch 'dev' into xbrl_meta_reshape
cmgosnell May 16, 2023
7dffac0
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell May 16, 2023
8db8836
Add two fixes for utility_plant_summary_ferc1
e-belfer May 16, 2023
b5be4f9
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 17, 2023
9c42881
preliminary migration of renaming of the xbrl metadata using transfor…
cmgosnell May 17, 2023
4c56344
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell May 17, 2023
5536cd3
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 24, 2023
b7b8c5c
Pin grpcio<1.55 due to pool thread bug
zaneselvans May 24, 2023
791271b
Fixes for utility_plant_summary and balance_sheet_assets tables
e-belfer May 24, 2023
73f3dd1
begin migration of metadata management upstream/to table transforms
cmgosnell May 24, 2023
1f47ef7
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell May 24, 2023
76321be
Rename income_statement_ferc1 data cols to dollar_value
zaneselvans May 24, 2023
b0ecd43
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
zaneselvans May 24, 2023
eade731
Rename utility_plant_value to ending_balance
zaneselvans May 25, 2023
ee1340e
Fix misspellings in several variable & function names.
zaneselvans May 25, 2023
378ddb1
clean up deuplicated xbrl_factoids pre metadata merge
cmgosnell May 26, 2023
bb18e89
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell May 26, 2023
fa2701d
migrate the source_table of subcomponent calcs
cmgosnell May 26, 2023
c107933
migrate the *current wip version* of the calc fixes into the table tr…
cmgosnell May 26, 2023
08eef8f
omigosh set/reset index
cmgosnell May 26, 2023
a4c0955
convert the deduplicator of calc components into pd based transforms
cmgosnell May 26, 2023
37ac763
Fix naming issue for EOR commercial/industrial columns
e-belfer May 29, 2023
e5563ae
Fix merge conflicts between working branches
e-belfer May 29, 2023
b76251c
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans May 29, 2023
0c366e5
First attempt at sub-total calc check with income_statement_ferc1, ne…
e-belfer May 29, 2023
bdcf080
migrate calcuation checks into the ferc1 table transformers
cmgosnell May 31, 2023
8dba9db
remove old cruft of process_xbrl_metadata
cmgosnell May 31, 2023
1b60bda
Merge branch 'check_calcs_in_transformers' into sub_total_xbrl_calc
e-belfer May 31, 2023
4aacfa0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 31, 2023
a99eda5
WIP Update to match migration of calc check
e-belfer May 31, 2023
719095e
Merge branch 'sub_total_xbrl_calc' of https://github.com/catalyst-coo…
e-belfer May 31, 2023
557959d
initial add of all of the income statement calc checks
cmgosnell May 31, 2023
b9a24ef
migrate the subcomonent removal to the vanilla long way
cmgosnell May 31, 2023
39be2e8
Merge branch 'xbrl_meta_reshape' into check_calcs_in_transformers
cmgosnell May 31, 2023
99a40e3
Merge branch 'check_calcs_in_transformers' into sub_total_xbrl_calc
e-belfer May 31, 2023
0db3169
Fix sub-total calc
e-belfer May 31, 2023
858a1b3
Merge branch 'sub_total_xbrl_calc' of https://github.com/catalyst-coo…
e-belfer May 31, 2023
a3d330f
Remove testing log statement
e-belfer May 31, 2023
05f5b89
fix calc cleaning oooops
cmgosnell Jun 1, 2023
319b83b
Add two tables and improve sub-total calc warning
e-belfer Jun 1, 2023
aa4f6c1
fix lil bugs in calc cleanings
cmgosnell Jun 1, 2023
0e5441c
Fix divide by nonetype
e-belfer Jun 1, 2023
0139908
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 1, 2023
a9ceec6
Merge branch 'xbrl_meta_reshape' into check_calcs_in_transformers
zaneselvans Jun 1, 2023
d32be5e
Add docs, change to calc subcomponents of reported values
e-belfer Jun 1, 2023
62089e2
Merge branch 'check_calcs_in_transformers' into sub_total_xbrl_calc
e-belfer Jun 1, 2023
87042b7
docs and name cleanup
cmgosnell Jun 1, 2023
7127067
Merge branch 'check_calcs_in_transformers' of github.com:catalyst-coo…
cmgosnell Jun 1, 2023
2ae2026
Merge branch 'check_calcs_in_transformers' into sub_total_xbrl_calc
e-belfer Jun 1, 2023
0e65ba7
convert single quote string to json & move check calc step to post be…
cmgosnell Jun 1, 2023
7885937
Fix references to XBRL metadata assets in FERC 1 notebook
zaneselvans Jun 1, 2023
738b9a1
fix dagster stuff for the glue tests. feed the dag the right assets d…
cmgosnell Jun 1, 2023
0edf88e
Merge branch 'check_calcs_in_transformers' into calc-correction-records
zaneselvans Jun 1, 2023
456605d
Apply json.dumps() directly.
zaneselvans Jun 1, 2023
e9f0203
Add records that correct overall calculation results in FERC 1.
zaneselvans Jun 2, 2023
49bc1c7
Use off_df as source for correction records.
zaneselvans Jun 2, 2023
f9229e1
Merge branch 'check_calcs_in_transformers' into sub_total_xbrl_calc
e-belfer Jun 2, 2023
d19d8b6
Merge pull request #2615 from catalyst-cooperative/sub_total_xbrl_calc
e-belfer Jun 2, 2023
fe4fc9e
Merge pull request #2618 from catalyst-cooperative/check_calcs_in_tra…
cmgosnell Jun 2, 2023
153b7e0
Merge branch 'xbrl_meta_reshape' into calc-correction-records
zaneselvans Jun 2, 2023
f3c7ed5
Rename check_table_calculations; fillna() in correction calcs
zaneselvans Jun 2, 2023
d09735c
Expand docstring for reconcile_table_calculations()
zaneselvans Jun 2, 2023
274acad
Use natural plant_in_service_ferc1 PK
zaneselvans Jun 2, 2023
f96ee5b
Merge pull request #2620 from catalyst-cooperative/calc-correction-re…
zaneselvans Jun 2, 2023
2f747a9
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 5, 2023
44e427e
cast calculations as pd.StringDtype()
zaneselvans Jun 5, 2023
c6e4679
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 5, 2023
b8b234e
Fix some spelling. Change calced to calculated.
zaneselvans Jun 6, 2023
fa240b5
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 6, 2023
d078968
add minor inter table calc fixes
cmgosnell Jun 6, 2023
9aae71e
respond to initial round of pr comments`
cmgosnell Jun 6, 2023
5d4244d
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 7, 2023
a2ae1ff
fix alembic haed thing and simplify bespoke metadata deduping
cmgosnell Jun 7, 2023
0be6511
Merge branch 'xbrl_meta_reshape' of github.com:catalyst-cooperative/p…
cmgosnell Jun 7, 2023
b762cc5
Merge branch 'xbrl_meta_reshape' into inter-table-calc-component-cleanup
cmgosnell Jun 7, 2023
595e5a0
migrate deduping to its own method
cmgosnell Jun 7, 2023
ae60989
migrate param info selection up into the param classes
cmgosnell Jun 7, 2023
e9eae69
Merge branch 'xbrl_meta_reshape' into inter-table-calc-component-cleanup
cmgosnell Jun 7, 2023
91d7eef
lingering cleaning post migrating the xbrl meta deduplication into a …
cmgosnell Jun 7, 2023
0fe2c62
oppsies swap back the right comment in the calc fixes
cmgosnell Jun 7, 2023
ce2b484
remove lil debugging loggers plus mor docs
cmgosnell Jun 7, 2023
794e827
Merge branch 'dev' into xbrl_meta_reshape
zaneselvans Jun 7, 2023
af7aef7
Merge branch 'xbrl_meta_reshape' into inter-table-calc-component-cleanup
zaneselvans Jun 7, 2023
a218eb2
Merge pull request #2635 from catalyst-cooperative/inter-table-calc-c…
cmgosnell Jun 8, 2023
0f72803
Merge branch 'dev' into xbrl_meta_reshape
cmgosnell Jun 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
22 changes: 4 additions & 18 deletions devtools/debug-ferc1-etl.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,6 @@
"pd.options.display.max_columns = None"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e21fea69-9d14-48d6-b0c9-149c37844e95",
"metadata": {},
"outputs": [],
"source": [
"logger = logging.getLogger()\n",
"logger.setLevel(logging.INFO)\n",
"handler = logging.StreamHandler(stream=sys.stdout)\n",
"formatter = logging.Formatter(\"%(message)s\")\n",
"handler.setFormatter(formatter)\n",
"logger.handlers = [handler]"
]
},
{
"cell_type": "markdown",
"id": "14145b03-fb79-46a6-a0e6-82da0c549d66",
Expand Down Expand Up @@ -128,10 +113,11 @@
"source": [
"from dagster import build_op_context\n",
"\n",
"from pudl.extract.ferc1 import xbrl_metadata_json\n",
"from pudl.extract.ferc1 import raw_xbrl_metadata_json\n",
"from pudl.transform.ferc1 import clean_xbrl_metadata_json\n",
"\n",
"context = build_op_context()\n",
"xbrl_metadata_json_dict = xbrl_metadata_json(context)"
"xbrl_metadata_json_dict = clean_xbrl_metadata_json(raw_xbrl_metadata_json(context))"
]
},
{
Expand Down Expand Up @@ -386,7 +372,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.10"
"version": "3.11.3"
}
},
"nbformat": 4,
Expand Down