Update dsra psra oct2021 (model-factory) #96

wkhchow · 2022-01-18T18:43:04Z

all layers renamed all_indicators to _indicators
all applicable layers with r2/b0 changed to r1/b0
DSRA
- revised indicators, and layers in line with newest dsra attributes https://github.com/OpenDRR/earthquake-scenarios/blob/master/FINISHED/geopackages/dsra_attributes_en.xlsx
PSRA
- revised indicators, and layers in line with newest psra attributes (xls not in github yet)
- build off newest canada-srm2, still missing BC province but code is there once BC data is ready

Fixes:

- added placeholder 'MMI' indicator - changed all_indicators to indicators

- added placeholder 'MMI' indicator - changed all_indicators to indicators - removed vs30lon, vs30lat - fixed prefix sD_Collapse to sDt_Collapse - added CSD aggregation level

- change r2 to r1

- changed all_indicators to indicators - renamed r2 to r1 indicators - added eq risk index calculations / indicators - removed commented out sections of code

- rename r2 to r1 - removed/reordered stat, loss_value from agg_curves_stats - added occupants to avg_loss_stats - removed trt from src_loss

- removed commented out codes - renamed r2 to r1 - add occupants, removed trt

- remove commented out lines

- remove commented out lines - changed all_indicators to indicators

- changed all_indicators to indicators

- remove commented out lines - changed all_indicators to indicators

…ste.sql - remove commented out lines - changed all_indicators to indicators

…e.sql - remove commented out lines - changed all_indicators to indicators

…e_ste.sql - remove commented out lines - changed all_indicators to indicators

- remove commented out lines - changed all_indicators to indicators

- remove commented out lines - changed all_indicators to indicators - remove sH_RupAbbr

- remove commented out lines - remove sh_rupabbr

- remove commented out lines - changed all_indicators to indicators

- remove commented out lines

- changed all_indicators to indicators - updated VHt_NonResHa

- remove commented out lines - changed r2 to r1

- remove commented out lines - changed all_indicators to indicators - changed r2 to r1

- comment out P/T not ready yet for initial testing

- Et_RESHD to Et_ResHD

- change all_indicators to indicators

update on_uhs to pe_uhs

added code to round indicators to 6 dp

jvanulde · 2022-01-19T17:12:51Z

scripts/PSRA_copyTables_Canada.py

+    args = parse_args()
+
+    # Copy agg curves stats b0 table
+    with open("/usr/src/app/ebRisk/Canada/ebR_Canada_agg_curves-stats_b0.csv".format(


We should avoid system paths.

Pass the path in as an argument?

Probably best.

Or maybe get the path from an environment variable, and with a default fallback if the environment variable is unset?
(Or maybe accept both command-line argument and environment variable?)

I took a brief look at it back in June 2021, and from what I recall, it is not just the Python scripts, but also some SQL files where the /usr/src/app path is hardcoded. See OpenDRR/opendrr-api#114, and the newly opened #98.

jvanulde

This looks fine to me. Eventually, it might be worth looking at ways to reduce the reliance on system paths for files if possible. I would like to to see the renovate.json configuration file removed and the requirements.txt updated to reflect current state. I ran some of the python scripts through Flake8 and there are unused imports in many files. Suggest that we enable Flake8 linter in VS Code.

anthonyfok

@wkhchow and @drotheram Wow! There are so many intricate details that it is nothing short of amazing how you meticulously keep track of them and make them work! Kudos!

drotheram

Big changes. Everything looks good to me. Note, PSRA results for B.C. are still not included in this pipeline yet

wkhchow added 30 commits October 5, 2021 11:50

Update Create_scenario_risk_building_Indicators_ALL_shkmp.sql

4e69f8c

- added placeholder 'MMI' indicator - changed all_indicators to indicators

Update Create_scenario_risk_sauid_Indicators_ALL_shkmp.sql

53e771e

- added placeholder 'MMI' indicator - changed all_indicators to indicators - removed vs30lon, vs30lat - fixed prefix sD_Collapse to sDt_Collapse - added CSD aggregation level

Update PSRA_combineSrcLossTable.py

c50b81a

- change r2 to r1

Update psra_4.Create_psra_sauid_all_indicators.sql

d50eb2e

- changed all_indicators to indicators - renamed r2 to r1 indicators - added eq risk index calculations / indicators - removed commented out sections of code

Update PSRA_copyTables.py

f8fdcfb

- rename r2 to r1 - removed/reordered stat, loss_value from agg_curves_stats - added occupants to avg_loss_stats - removed trt from src_loss

Update psra_2.Create_table_updates.sql

b3cf9cb

- removed commented out codes - renamed r2 to r1 - add occupants, removed trt

Update Create_all_tables_update.sql

674dee8

- remove commented out lines

Update Create_hexbin_MH_risk_sauid_prioritization_aggregation_area.sql

9c0739d

- remove commented out lines - changed all_indicators to indicators

Update Create_hexbin_physical_exposure_aggregation_area_proxy.sql

61540fc

- remove commented out lines - changed all_indicators to indicators

Update Create_hexbin_social_vulnerability_aggregation_area_proxy.sql

5c6e84e

- changed all_indicators to indicators

Update Create_MH_risk_sauid_prioritization_Canada.sql

ef7af38

- changed all_indicators to indicators

Update Create_physical_exposure_building_indicators_PhysicalExposure.sql

c288153

- remove commented out lines - changed all_indicators to indicators

Update Create_physical_exposure_building_indicators_PhysicalExposure_…

99d190a

…ste.sql - remove commented out lines - changed all_indicators to indicators

Update Create_physical_exposure_sauid_indicators_view_PhysicalExposur…

4471360

…e.sql - remove commented out lines - changed all_indicators to indicators

Update Create_physical_exposure_sauid_indicators_view_PhysicalExposur…

4241185

…e_ste.sql - remove commented out lines - changed all_indicators to indicators

Update Create_physical_exposure_site_level_indicators_PhysicalExposur…

c0630dd

…e_ste.sql - remove commented out lines - changed all_indicators to indicators

Update Create_risk_dynamics_indicators.sql

78dd6f4

- remove commented out lines - changed all_indicators to indicators

Update Create_scenario_risk_building_Indicators_ALL_ste_shkmp.sql

f16e878

- remove commented out lines - changed all_indicators to indicators - remove sH_RupAbbr

Update Create_scenario_risk_master_tables.sql

083fb58

- remove commented out lines - remove sh_rupabbr

Update Create_scenario_risk_sauid_Indicators_ALL_ste_shkmp.sql

521f1d1

- remove commented out lines - changed all_indicators to indicators

Update Create_scenario_risk_site_level_Indicators_ALL_ste_shkmp.sql

fac7438

- remove commented out lines - changed all_indicators to indicators

Update Create_site_exposure_to_building_and_sauid.sql

bc36bf7

- remove commented out lines

Update Create_social_vulnerability_sauid_indicators_SocialFabric.sql

8b320ca

- changed all_indicators to indicators - updated VHt_NonResHa

Update psra_1.Create_tables.sql

55560c1

- remove commented out lines - changed r2 to r1

Update psra_3.Create_psra_building_all_indicators.sql

b9545bb

- remove commented out lines - changed all_indicators to indicators - changed r2 to r1

Update psra_5.Create_psra_sauid_references_indicators.sql

d6d48f9

- remove commented out lines - changed all_indicators to indicators - changed r2 to r1

Update psra_6.Create_psra_merge_into_national_indicators.sql

e84aa91

- comment out P/T not ready yet for initial testing

Update Create_hexbin_physical_exposure_aggregation_area_proxy.sql

0c72b62

- Et_RESHD to Et_ResHD

Update psra_6.Create_psra_merge_into_national_indicators.sql

af327f6

- change all_indicators to indicators

Update psra_6.Create_psra_merge_into_national_indicators.sql

7014f47

update on_uhs to pe_uhs

wkhchow added 3 commits January 11, 2022 12:23

psra eqri update

7021d1e

added code to round indicators to 6 dp

sC_Empl_* to sC_Empl*

5a85405

sCr_Empl90 to sCt_Empl90

7b0d3d2

wkhchow added this to the Sprint 50 milestone Jan 18, 2022

wkhchow requested review from anthonyfok, jvanulde, drotheram and DamonU2 January 18, 2022 18:43

wkhchow assigned drotheram and wkhchow Jan 18, 2022

wkhchow mentioned this pull request Jan 18, 2022

Update dsra psra oct2021 (opendrr-api) OpenDRR/opendrr-api#154

Merged

wkhchow changed the title ~~Update dsra psra oct2021~~ Update dsra psra oct2021 (model-factory) Jan 18, 2022

jvanulde reviewed Jan 19, 2022

View reviewed changes

jvanulde approved these changes Jan 19, 2022

View reviewed changes

anthonyfok mentioned this pull request Jan 19, 2022

Investigate if we could de-hardcode /usr/src/app #98

Open

update eqri normalized scores from 0 to 1 to 0 to 100

6710c72

drotheram linked an issue Jan 21, 2022 that may be closed by this pull request

PSRA - add aggregation column for agg_curves_stats, agg_losses_stats, src_loss_table from filename #84

Closed

DamonU2 approved these changes Jan 21, 2022

View reviewed changes

anthonyfok mentioned this pull request Jan 21, 2022

[testing] Update dsra psra oct2021 (opendrr-api) anthonyfok/opendrr-api#5

Merged

anthonyfok approved these changes Jan 21, 2022

View reviewed changes

anthonyfok force-pushed the update_dsra_psra_oct2021 branch from 6f2ae3a to 6710c72 Compare January 21, 2022 21:00

anthonyfok merged commit 6710c72 into master Jan 21, 2022

drotheram reviewed Jan 21, 2022

View reviewed changes

wkhchow deleted the update_dsra_psra_oct2021 branch April 1, 2022 21:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dsra psra oct2021 (model-factory) #96

Update dsra psra oct2021 (model-factory) #96

wkhchow commented Jan 18, 2022 •

edited by anthonyfok

Loading

jvanulde Jan 19, 2022

drotheram Jan 19, 2022

jvanulde Jan 19, 2022

anthonyfok Jan 19, 2022

jvanulde left a comment

anthonyfok left a comment

drotheram left a comment

Update dsra psra oct2021 (model-factory) #96

Update dsra psra oct2021 (model-factory) #96

Conversation

wkhchow commented Jan 18, 2022 • edited by anthonyfok Loading

jvanulde Jan 19, 2022

Choose a reason for hiding this comment

drotheram Jan 19, 2022

Choose a reason for hiding this comment

jvanulde Jan 19, 2022

Choose a reason for hiding this comment

anthonyfok Jan 19, 2022

Choose a reason for hiding this comment

jvanulde left a comment

Choose a reason for hiding this comment

anthonyfok left a comment

Choose a reason for hiding this comment

drotheram left a comment

Choose a reason for hiding this comment

wkhchow commented Jan 18, 2022 •

edited by anthonyfok

Loading