Multiple cycle upload #4073

anchapin · 2023-06-02T14:53:23Z

Any background context you want to provide?

What's this PR do?

Allows you to upload a spreadsheet with data across multiple cycles.

How should this be manually tested?

Upload the file 'Sample_MultipleCycles.xlsx' and select the 'Multiple Cycle Upload' checkbox.

Additionally, ensure that the SEED column 'year_ending' is mapped to the "Year Ending" column from the 'Date File Header'.

What are the relevant tickets?

#3960 #4053 #3181

Screenshots (if appropriate)

…orm/seed into multiple_cycle_upload

seed/data_importer/match.py

seed/static/seed/js/controllers/data_upload_modal_controller.js

haneslinger · 2023-06-02T17:29:45Z

seed/data_importer/tasks.py

+        celery_chain(
+            _geocode_properties_or_tax_lots.si(file_pk, progress_data.key),
+            group(_map_additional_models.si(id_chunk, file_pk, progress_data.key) for id_chunk in id_chunks),
+            match_and_link_incoming_properties_and_taxlots.si(file_pk, progress_data.key, sub_progress_data.key),
+            finish_matching.s(file_pk, progress_data.key),
+        )()
+
+        sub_progress_data.total = 100
+        sub_progress_data.save()
+
+        return {'progress_data': progress_data.result(), 'sub_progress_data': sub_progress_data.result()}


This part and it's counterpart above can be moved out of the if/else

Really? How would that work? Lines 1355 and 1380 are different and I didn't think it would work if they were moved out of the if/else.

haneslinger · 2023-06-02T17:30:43Z

seed/data_importer/tasks.py

@@ -1580,7 +1620,7 @@ def _get_field_from_obj(field_obj, field):


 @shared_task
-def _map_additional_models(ids, file_pk, progress_key):
+def _map_additional_models(ids, file_pk, progress_key, cycle=None):


Idk if this whole function needs to be branched like this

What would you suggest? Should I make a helper function again like the match_and_link_incoming_properties_and_taxlots function in 'match.py'?

haneslinger · 2023-06-02T17:31:27Z

seed/data_importer/tasks.py

+        id_chunks = [[obj.id for obj in chunk] for chunk in batch(ids, 100)]
+
+        progress_data.total = (
+            1  # geocoding
+            + len(id_chunks)  # map additional models tasks
+            + 2  # match and link
+            + 1  # finish
+        )
+        progress_data.save()
+
+        for id_chunk in id_chunks:
+            property_states = PropertyState.objects.filter(id__in=id_chunk).prefetch_related('building_files')


I don't think these chunks are doing anything

These chunks were existing and I wasn't sure how to check whether it was doing anything or not. Any suggestions?

haneslinger · 2023-06-02T18:46:55Z

Just documenting a known problem: Should a property's year ending not fit into any existing cycles, the API 500s and the UI hangs.

haneslinger · 2023-06-02T18:53:52Z

idk if this is intentional, but If you selected multi-cycle and a cycle, it will only upload the properties that are part of the selected cycle

>>> PropertyView.objects.values("cycle__name")
<QuerySet [{'cycle__name': '2018 Calendar Year'}]>

…orm/seed into multiple_cycle_upload

haneslinger

sweet!

seed/data_importer/tasks.py

…orm/seed into multiple_cycle_upload

anchapin · 2023-06-30T03:14:51Z

Note that tax lots don't work (in that they only get added to the first cycle listed in the upload file) but it doesn't break SEED. This seems sufficient for now since the first known customer for this feature indicated that they aren't planning to use tax lots initially.

There is an issue with "Year Ending" just being a year value like "2019" though. It won't find a matching cycle and always use the default cycle.

from datetime import date, datetime to match other test files. Also added spreadsheet to match the test

Added date verification to cycle admin page

* getting close to mvp just squashing bugs * Fixed the bugs and have mvp ready * Anonomized data * Renamed sample file * Pass celery state ids instead of states * Resolve a review comment on the modal code * Clean * Fixes the hang issue * unittest for multi cycle upload * cleanup * Cycle conversion to date * update test for edge cases * update cycle filter * Update to multiple_cycle_upload changes * Updated cycle filter again * Finished other than modal bug * Modal bug is fixed * year ending using date over datetime * precommit * cycle start and end to date from datetime api test * add test for cycle spanning a typical calendar year * test formatting for clarity * precommit * cycle start and end to date from datetime unit test * cycle date formatting without timezone * Fixed the blank year_ending case and PR comments * precommit * revert cycle start and end to datetime * convert year ending to awawre datetime * add help text to multicycle checkbox * translations * add an info icon hover to display help text * Migration * Fixed issues for release * Migration conflict * Minor change to a comment * Switched to import datetime instead of from datetime import date, datetime to match other test files. Also added spreadsheet to match the test * Converted cycle datetime fields to date fields Added date verification to cycle admin page * Improved model * Performance improvement * Consistent date usage in tests --------- Co-authored-by: Hannah Eslinger <heslinge@nrel.gov> Co-authored-by: Ross Perry <perryr16@gmail.com> Co-authored-by: Alex Swindler <Alex.Swindler@nrel.gov>

RDmitchell · 2023-07-28T01:18:26Z

@anchapin -- do the all the cycles that correspond to the Year Ending field need to be defined in order for the records to get associated with those cycles?

RDmitchell · 2023-07-28T01:23:00Z

@anchapin -- also, in this test file, is the first record, with year ending set to 1/1/2021, supposed to be part of the 2020 cycle? or a 2021 cycle?

RDmitchell · 2023-07-28T01:27:15Z

@anchapin -- does it matter what the default cycle is set to when the Multiple Cycle box is checked?

RDmitchell · 2023-07-28T01:29:22Z

@anchapin -- I think never mind to the default cycle question -- it appears that is the cycle used if the Year Ending field is blank.

RDmitchell · 2023-07-28T01:33:33Z

@anchapin -- so I guess the date in the Year Ending field just has to be equal to or between the dates defined in the cycle, right? so the record with year ending of 1/1/2021 would be associated the the 2021 cycle if the cycle were defined as 01/01/2021 - 12/31/2021, correct? That seems to be the way the program is interpreting it. Just need to make sure I understand the mechanics so I can document it. Thx

anchapin added 3 commits June 1, 2023 22:32

getting close to mvp just squashing bugs

a059eca

Fixed the bugs and have mvp ready

a7ec776

Anonomized data

8f9f581

anchapin added the Feature Add this label to new features. This will be reflected in the change log when generated. label Jun 2, 2023

anchapin requested review from axelstudios and haneslinger June 2, 2023 14:53

anchapin added 3 commits June 2, 2023 10:53

Merge branch 'develop' into multiple_cycle_upload

195821e

Renamed sample file

f9d22be

Merge branch 'multiple_cycle_upload' of https://github.com/SEED-platf…

e0612f1

…orm/seed into multiple_cycle_upload

haneslinger requested changes Jun 2, 2023

View reviewed changes

Pass celery state ids instead of states

d0c38f9

anchapin and others added 4 commits June 6, 2023 09:25

Resolve a review comment on the modal code

9f3050c

Clean

4f02608

Fixes the hang issue

5edcb5c

Merge branch 'multiple_cycle_upload' of https://github.com/SEED-platf…

dc35b93

…orm/seed into multiple_cycle_upload

haneslinger approved these changes Jun 8, 2023

View reviewed changes

seed/data_importer/tasks.py Outdated Show resolved Hide resolved

perryr16 and others added 12 commits June 8, 2023 15:42

unittest for multi cycle upload

5a8bbde

cleanup

d216b75

Cycle conversion to date

e5122c9

update test for edge cases

acf7de8

update cycle filter

1a55f75

Update to multiple_cycle_upload changes

3b4fb94

Merge branch 'multiple_cycle_upload' of https://github.com/SEED-platf…

6d2816f

…orm/seed into multiple_cycle_upload

Updated cycle filter again

b249033

Finished other than modal bug

ecb560d

Modal bug is fixed

13fcde1

year ending using date over datetime

b2c5a30

precommit

42a71fc

perryr16 and others added 9 commits June 13, 2023 15:49

Merge branch 'multiple_cycle_upload' of https://github.com/SEED-platf…

a94b4a4

…orm/seed into multiple_cycle_upload

precommit

f6923bd

revert cycle start and end to datetime

3d20338

convert year ending to awawre datetime

b33c1f1

add help text to multicycle checkbox

5150b6b

translations

cdbf91d

add an info icon hover to display help text

15b0b0c

Migration

4cafc67

Fixed issues for release

09a1c39

anchapin added 4 commits June 29, 2023 23:14

Merge branch 'develop' into multiple_cycle_upload

9f656f4

Migration conflict

070c524

Minor change to a comment

2ef9272

Switched to import datetime instead of

481c29c

from datetime import date, datetime to match other test files. Also added spreadsheet to match the test

axelstudios force-pushed the multiple_cycle_upload branch 2 times, most recently from b5870f3 to 0e5cf14 Compare July 3, 2023 17:09

Converted cycle datetime fields to date fields

7b6e007

Added date verification to cycle admin page

axelstudios force-pushed the multiple_cycle_upload branch from 0e5cf14 to 7b6e007 Compare July 3, 2023 17:39

axelstudios added 2 commits July 3, 2023 12:34

Improved model

603a3a6

Performance improvement

9adfb14

axelstudios approved these changes Jul 3, 2023

View reviewed changes

Consistent date usage in tests

96e2782

axelstudios merged commit 41fa04b into develop Jul 3, 2023
7 checks passed

axelstudios deleted the multiple_cycle_upload branch July 3, 2023 21:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple cycle upload #4073

Multiple cycle upload #4073

anchapin commented Jun 2, 2023 •

edited

haneslinger Jun 2, 2023

anchapin Jun 6, 2023

haneslinger Jun 2, 2023

anchapin Jun 6, 2023

haneslinger Jun 2, 2023

anchapin Jun 6, 2023

haneslinger commented Jun 2, 2023

haneslinger commented Jun 2, 2023

haneslinger left a comment

anchapin commented Jun 30, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

Multiple cycle upload #4073

Multiple cycle upload #4073

Conversation

anchapin commented Jun 2, 2023 • edited

Any background context you want to provide?

What's this PR do?

How should this be manually tested?

What are the relevant tickets?

Screenshots (if appropriate)

haneslinger Jun 2, 2023

Choose a reason for hiding this comment

anchapin Jun 6, 2023

Choose a reason for hiding this comment

haneslinger Jun 2, 2023

Choose a reason for hiding this comment

anchapin Jun 6, 2023

Choose a reason for hiding this comment

haneslinger Jun 2, 2023

Choose a reason for hiding this comment

anchapin Jun 6, 2023

Choose a reason for hiding this comment

haneslinger commented Jun 2, 2023

haneslinger commented Jun 2, 2023

haneslinger left a comment

Choose a reason for hiding this comment

anchapin commented Jun 30, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

RDmitchell commented Jul 28, 2023

anchapin commented Jun 2, 2023 •

edited