This repository was archived by the owner on Jun 30, 2023. It is now read-only.
Conversation
1. fixing inactive logic, changing x_inactive to job_inactive 2. fixing qaqc statements in qaqc_init.sql
Closed
mgraber
reviewed
Jun 16, 2020
Comment on lines
+268
to
+282
| -- -- x_inactive | ||
| -- WITH CORR_target as ( | ||
| -- SELECT a.job_number, | ||
| -- COALESCE(b.reason, 'NA') as reason, | ||
| -- b.edited_date | ||
| -- FROM STATUS_devdb a, housing_input_research b | ||
| -- WHERE a.job_number=b.job_number | ||
| -- AND b.field = 'x_inactive' | ||
| -- AND (upper(a.x_inactive)=upper(b.old_value) | ||
| -- OR (a.x_inactive IS NULL | ||
| -- AND (b.old_value IS NULL | ||
| -- OR b.old_value = 'false'))) | ||
| -- ) | ||
| -- UPDATE CORR_devdb a | ||
| -- SET x_dcpedited = array_append(x_dcpedited, 'x_inactive'), | ||
| -- x_reason = array_append(x_reason, json_build_object( | ||
| -- 'field', 'x_inactive', 'reason', b.reason, | ||
| -- 'edited_date', b.edited_date | ||
| -- )) | ||
| -- FROM CORR_target b | ||
| -- WHERE a.job_number=b.job_number; | ||
|
|
||
| UPDATE STATUS_devdb a | ||
| SET x_inactive = trim(b.new_value) | ||
| FROM housing_input_research b | ||
| WHERE a.job_number=b.job_number | ||
| AND b.field = 'x_inactive' | ||
| AND a.job_number in ( | ||
| SELECT DISTINCT job_number | ||
| FROM CORR_devdb | ||
| WHERE 'x_inactive'=any(x_dcpedited)); No newline at end of file | ||
| -- UPDATE STATUS_devdb a | ||
| -- SET x_inactive = trim(b.new_value) | ||
| -- FROM housing_input_research b | ||
| -- WHERE a.job_number=b.job_number | ||
| -- AND b.field = 'x_inactive' | ||
| -- AND a.job_number in ( | ||
| -- SELECT DISTINCT job_number | ||
| -- FROM CORR_devdb | ||
| -- WHERE 'x_inactive'=any(x_dcpedited)); No newline at end of file |
Contributor
There was a problem hiding this comment.
Should these be uncommented or removed?
Contributor
Author
There was a problem hiding this comment.
unclear, because x_inactive needs to be changed in the research table first, and there are also corresponding qaqc checks, so I decided to comment it for now and address it during QAQC
51 tasks
SPTKL
added a commit
that referenced
this pull request
Jun 24, 2020
* Refactor baiyue (#84) * first commit * adding more scripts * adding co table files * adding status * refactoring geocoding and assigning geometry * fixes with uid * update uid * fixing logic for CO table and address spatial join * adding header comments * adding logic for handling multiple CO assignment * fixing _geo.sql * create corr_devdb * fixing occ * Refactor hny (#82) * Use select statements to find matches * Corrent not-equal syntax and filter out all demolitions * Identify many-to-one and one-to-many * Remove right join between minimum match proprity and all matches * Add one to one and multiple hny logic * Add many-to-one and combine into a single lookup * Retain job_numbers that are not min for many-to-one * Add occ category to many-dev-one-hny * Comment complicated many-to-many logic * Remove duplicate many-to-many cases * Apply logic to one dev many hny without other filter * Reformat comments * Remove one dev many hny filter from min job table * Fix typo where total_units got all_counted_units * Add header and match devdb schema names * Merge with devdb * Add corrections procedure * rename dev_hny_lookup to HNY_lookup where necessary * Drop hny_matches if exists * Add hny_jobrelate * Filter matches to highest priority in both directions * adding ownername in devdb_Create_init.sql (#86) * cleaning address number remove rear, gar, garage and leading 0 (#87) * 38 recode status (#88) * cleaning address number remove rear, gar, garage and leading 0 * recoding status based on new status lookup #38 * Append hotel and class b fields from corrections (#89) * assigning quarter complete, quater permit using the same logic as year_complete an year_permit #51 * issue #50 field recodes * issue #50 field recodes * issue #27 assigning occ_category as Residential based on hotel units and otherb units in MID_devdb * adding new status codes to downstream processes * Only populate complete dates for records with complete status (#96) * Set year and quarter complete to null based on status * Fix partial complete capitalization * Only assign complete year to complete status * merge Enlrg_Horiz and Enlrg_Vert to enlargement * 50 recode fields (#92) * issue #50 field recodes * issue #50 field recodes * merge Enlrg_Horiz and Enlrg_Vert to enlargement * Create temp _MID, then create MID with status fields (#98) * Set year and quarter complete to temp before status workflow * Rename devdb_create_mid to make _MID_devdb * Mark status q and co year/qtr complete as temp * Update hny_id and input table (#99) * Update hny_id and input table * Output project_id to HNY_matches for stability research * renaming scripts, and correcting table names * Type cast dates, first filtering invalid ones (#103) * Typecast dates * Check for valid dates before typecasting * Check for valid dates using function * move is_date.sql to bash script * remove address cleaning * general refactor of corrections process and turn spatial join into functions * 1. adding more spatial boundaries (dcp_puma, dcp_firecompany, doe_eszones, doe_mszones 2. adding corresponding spatial join functions to _function.sql 3. implementing _spatil.sql so that for addresses in tpad, take spatial boundaries from spatial join 4. adding additional spatial attributes in parser in geocode.py (geo_puma, geo_firedivision, geo_firebattalion, geo_firecompany) 5. still pending confusion about centroid assignment, waiting on HED inputs 6. around 1460 records without a geom * adding ntaname and nta_translate function * 1. updating geom assignment logic so that 1. taking geom from geosupport 2. take bin centroid from building footprints using dob bin 3. taking bbl centroid from building footprints using dob bbl 4. taking bbl centroid from mappluto using geosupport bbl 5. taking bbl centroid from mappluto using dob bbl 6. taking bbl centroid from DTM using dob bbl * Add corrections procedure to hny * Fix typo and remove hny_no_matches if exists * 104 date qaqc (#108) * Switch date fields to new schema * Change schema for date fields downstream * Correct dates and output qaqc for invalid * Remove old_value check for date corrections * Change affordable_units to units_hnyaff * uncomment * adding function to assign non residential flag * changing downstream for resid_flag and nonres_flag * status_q -> date_permittd * Switch date correction to new dcpedited format (#111) * Add qaqc folder * Move date cast (#112) * Move date cast to init.sql * Fix _init to init typo * Add qaqc for test records * Name working QAQC tables by step * Simplify qaqc table naming conventions * (#35) assigning inactive flag in _status.sql (#116) 1. fixing inactive logic, changing x_inactive to job_inactive 2. fixing qaqc statements in qaqc_init.sql * Initial schema changes (#113) * fixing missing schema changes * fixing geom assignment order based on (#79) (#115) * fixing geom assignment order based on (#79) also fixing corrections in _init.sql * update geom order * Fix parentheses * QAQC checks for units step (#117) * Add units-related QAQC checks * Change field names for qaqc bools * Remove checks relying on bbl * Drop if exists and set input to occ * Remove duplicate checks from comments * Remove checks depending on resid_flag * Change input to reflect occ checks in mid * QAQC occupancy checks (#118) * Add occ qaqc * Move occupancy checks to MID * Add units checks that depend on resid_flag * Update description * Use MID for all checks, and UNITS for input qaqc * Remove qaqc_co.sql and qaqc_occ.sql * Fix job_desc and unit schema changes and close paren (#121) * Making sure a demolition with a non null date_Statux will get status 5.complete (#119) (#40) * QAQC for status fields (#122) * Change remaining x_inactive to job_inactive * Add z_inactive_with_update check and calls from sh * Change input qaqc to UNITS_qaqc * Add units_co_prop_mismatch and z_incomp_tract_home * Fix field names in units_co_prop_mismatch and z_incomp_tract_home * inactive_job -> job_inactive * Fix capture date env var * Update 02_build_devdb.sh Co-authored-by: SPTKL <caobaiyue@gmail.com> * Add job number for potential duplicates * Add flag for dem and nb sharing bin and bbl * Switch to spatial match for dem_nb_overlap * Switch spatial match to geo_bbl match * Switch back to bool and create table of matches * Reformat match tables * assign resid_flag solely based on units fields (#126) * Use geo_bbl for potential duplicate match * Remove duplicate matches * 79 schools spatial (#125) * Making sure a demolition with a non null date_Statux will get status 5.complete (#40) * adding elementry, middle school districts and school subdistricts * Update _function.sql * Resid_flag, dcpeditfields, remove x_mixeduse (#127) * Remove x_mixeduse, owner_firstnm & owner_lastnm, rename x_reason as dcpeditfields * Remove x_mixeduse corrections, fix typo in geom correction * Remove x_mixeduse from comments * x_reason -> dcpeditfields * Remove x_mixeduse and automatic change of hotel-> res and res -> hotel * Add nonres_flag to comments * x_reason -> dcpeditfields * x_reason -> dcpeditfields * x_reason -> dcpeditfields Co-authored-by: SPTKL <caobaiyue@gmail.com> * consolidate dataloading (#130) * consolidate dataloading * adding occ_lookup and status_lookup * fixing * Revise b_likely_occ_desc (#132) * blacken python files * Rework duplicate flags * fix * 128 qaqc format (#135) * Add script to alphabetize qaqc fields * Alter comments to track when fields get created * first round of clean up (#136) (#138) * first round of clean up (#136) * more clean up * adding historical building footprints * remove mappluto row limit * consolidate hny workflow and other shell scripts * adding housing_input_hny_job_manual back * Create FINAL_devdb (#139) * Only include hny relevant fields in HNY_devdb * Merge MID with hny & pluto and reorder fields * Add geo-fields * Add coalesce to take geosupport fields where possible * Use env var to set version * Fix permited status label * Change boro to code in init * Fix mislabel of status-q as a1-nb complete dates * Remove zoning sft fields from export * (#134) checking if point is in bbl before correction (#145) * (#134) checking if point is in bbl before correction * Update 02_build_devdb.sh * Update 02_build_devdb.sh * adding files to dataloading * Allow NULL-NULL BBL matches for duplicate checks (#152) * [weekly] * [weekly] * Update weekly.yml * 63 patch (#159) * (#63) change geom assignment order * (#63) change geom assignment order * adding dob lon lat to reduce nulls * long to lon * (#157) geospatial qaqc (#158) * adding docsify support (#162) Co-authored-by: Molly Graber <molly.graber@me.com>
SPTKL
added a commit
that referenced
this pull request
Apr 29, 2021
SPTKL
added a commit
that referenced
this pull request
Apr 29, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#35 this pull is replacing #90 (too out of sync)