Add UBID relation to state

[perryr16]

Any background context you want to provide?

The PropertyState (and TaxLotState) both have a ubid field that aligns with a canonical column. Per some new guidelines States can have multiple ubids. To accommodate this a new model UbidModel has been created with a many-to-one relationship with PropertyState and TaxlotState. Note that the name of the new model could not be ubid to prevent a naming conflict with the existing field.

A UbidModel will have the following fields:

• ubid: string
• preferred: boolean
• property: ForeignKey
• taxlot: ForeignKey

PNNL has developed a number of sql functions that can be migrated into a postgres db. Reference. Some of these functions allow a user to calculate the overlap of 2 unique UBID's. This is refered to as the Jaccard Index and the functions returns a value from 0 to 1 representing the match quality.

Now that UBIDs can be compared via the jaccard index, the matching and merging processes must be updated to use a threshold value instead of an exact match.

What's this PR do?

• Creates a relationship between -State and UbidModel. -States will still have a ubid column representing the 'prefered' ubid model
• Adds backend CRUD endpoints
• Adds SQL migration to the DB to leverage PNNL functions. The PNNL Function we are most interested in is UBID_CodeArea_Jaccard. This function has a number of dependent functions required to run. These dependant functions are: UBID_Decode, UBID_Parse, UBID_CodeArea, UBID_ValidateLatitude, UBID_ValidateLongitude, pluscode_isfull, pluscode_isvalid, pluscode_isshort, pluscode_decode, pluscode_codearea
• Generates a new Organization setting for a UBID threshold. The threshold is an org setting that must be between 0 and 1.0. If the jaccard index between an incoming ubid and an existing ubid is above the UBID threshold it will qualify as a match (if the other matching criteria are met)
• Uses signals to generate UbidModels when a new property or taxlot is uploaded with a ubid
• Uses signals to update -State.ubid if a saved UbidModel's "preferred" field is set to true
• Adds frontend CRUD. A new ubid viewer and editor are available from the inventory list actions, inventory detail actions, or under a new UBID inventory detail nav button. The structure is loosley based on the derived column admin and editors. New or updated ubids are validated before they may be saved.
• Adds a ubid jaccard comparison modal to the inventory list actions. It can be triggered when 0-2 properties are selected. If fewer than 2 properties are selected a user may use custom ubids for comparison.
• Renames the ubid_modal to ubid_decode_modal, which allows users to decode a property's ubid
• Allows properties to be merged together while preserving the ubid history.
• Adds a map view to the inventory detail and inventory detail ubid pages highlighting the Properties ubid bounding box. If there is no preferred ubid the map is disabled. If the preferred ubid is updated the page and map reload.

How should this be manually tested?

Run tests, test for functionality.

Possible testing procedure:

• upload a few properties with ubids
• set org setting ubid threshold to something small like 0.1 to allow matching
• upload the same properties with slightly different UBIDs (ex: 85FPPRR9+3C-0-0-0-0 ---> 85FPPRR9+3C-0-0-0-1)
• properties and ubids should have been merged with the preferred set to the incoming
• view Inventory Detail UBIDs page, verify map
• change the preferred UBID
• map should update
• create a UBID, delete a UBID
• set org setting ubid threshold to something high like 0.9 to prevent matching
• upload the same properties with different UBIDS (ex: 85FPPRR9+3C-0-0-0-1 ---> 85FPPRR9+3C-0-0-0-5)
• property list should have created new properties as the ubid threshold was not met
• set org setting ubid threshold to something small like 0.1
• upload same properties with different UBIDs
• existing properties should have been merged into the incoming properties
• ensure ubid history is preserved on Inventory Detail UBIDs
• merge 2 properties together, ensure ubid history
• unmerge property, check ubid history

What are the relevant tickets?

#3957
#4051
#4069
#4041
#4064

Screenshots (if appropriate)

Should this page include a graphic illustrating what a jaccard match looks like (as seen in PR background)?

[perryr16]

This migration could be moved into 0196

[perryr16]

I ran into a few errors attempting to import, read, and execute an entire SQL file. I'd like to refactor this to leverage imports, but I didn't want to let it become a blocker.

[axelstudios]

@perryr16 I fixed TONS of issues in this PR:

• Major improvements and bug fixes to the UBID editor and admin pages
• Major simplification & streamlining of the UBID comparison dialog
• Major improvements to functionality that was being called by both signal handlers and create/update functions
• Complete overhaul of all Swagger endpoints, and fixed incorrect documentation
• Reordered migrations
• Included original SQL scripts to retain licenses and greatly simplify future updates
• Fixed UBID threshold (it could be set to a value outside of the 0-1 range via the API)
  • Fixed 500 errors related to setting bad ubid threshold values
  • Changed the input to be numeric instead of text
• Fixed handling when ubid_threshold is zero, removed option for ubid_threshold to be null
• Fixed incorrect detail page nav titles (UBID page showed Tax Lot Detail Notes, etc)
• Fixed bad API response that didn't actually return the errors: {"status": "failed", "errors": "serializer.errors"}
• Fixed bug that allowed users to change UBIDs for records outside of their organization
• Fixed bug that allowed creation of a UbidModel for a property and taxlot combined
• Fixed bug where directly editing the UBID on a Property Detail page wouldn't update the preferred UBID
  • Fixed bug where directly editing the UBID to an empty string wouldn't update the UBID as non-preferred
• Fixed bug where marking the preferred UBID as non-preferred did not remove the UBID from the PropertyState
• Fixed bug where if an initially-imported record had multiple UBIDs, and then the record was directly edited on the detail page the non-preferred ubids wouldn't exist on the new state
• Fixed bug that allowed duplicate ubids to be added to a property or taxlot via unique constraints via two partial indexes to the ubidmodel
• Fixed bug where the property/taxlot id could be changed
• Removed unimplemented pagination functionality from ubid API
• Restricted the ability to associate UBIDs with initial DATA_STATE_IMPORT records
• Removed PATCH ubid endpoint
• Added client-side UBID validation via ubid directive for form inputs
• Added a new ModelViewSetWithoutPatch viewset
• Added a migration to rename the SEED column (not TaxLotState column) from ulid to ubid
• Added a migration to rehash all Property and TaxLot states
• Added a manage.py rehash command for testing
• Fixed inventory detail nav ids
• Fixed org nav sorting
• Added autofocus to Create UBID field input
• Fixed countless other 500 errors

Remaining Issues & Questions

• It would be nice to reload the map without reloading the state of the parent page
• Should changing the UBID threshold re-compare all records with UBIDs?
• Do we even need the ubidmodel.preferred column if we can just infer the preferred one from the state.ubid value?
• If UBID is not a matching field, and has merge-protection enabled, then importing a matching record with a different UBID won't create a non-preferred UBID and the information will be lost
• Identical invalid UBIDs still match, I guess that's the expected behavior

[RDmitchell]

@perryr16 -- I might need some help in terms of documenting this feature.

But to start, when you are in the Property Detail view and you click on the UBIDs link, there is a Create UBID button.

It looks like you already have to know the UBID for the property, right? SEED isn't actually generating the UBIDs, is it?

And then you can enter multiple UBIDs and then I guess SEED determines which is the "preferred" UBID?

[axelstudios]

@RDmitchell Yes, that Create UBID button assumes you already know a valid UBID. SEED won't currently take a property_footprint or lat/long and turn that into a UBID, but it could.

When you're on that UBID tab you can edit each one individually and select one as preferred (changing the preferred UBID automatically unsets a previously-preferred UBID and updates the ubid value for the property/taxlot