Parallelize snapshot creation #61

dav3r · 2021-01-12T19:19:33Z

🗣 Description

This PR modifies the weekly reporting script so that snapshot creation is multithreaded, instead of single-threaded.

While I was making changes to this script, I took the opportunity to clean up some function names, variable names, comments, and logging statements to make them clearer and more consistent (69795cc, fd426e5, d208d94). I also removed some old testing code that had been commented-out, but is no longer needed (70ea90c).

Resolves #58.

💭 Motivation and Context

From #58:

Creating snapshots is the biggest chunk of our weekly reporting time (about 700 minutes out of 850 minutes total; that's over 80% of the total time and climbing with each new report recipient we add). By creating multiple snapshots at once, we should be able to significantly reduce the total time it takes to create snapshots, just like we have done with report creation.

Since we have to pause the commander while snapshots and reports are generated, shorter reporting time means more weekly uptime for CyHy processing.

This PR starts us off with 16 threads for snapshot creation, since that is how many we currently use for report creation. That number can be increased or decreased in Production after we observe how long the threaded snapshot creation process takes and if there are any database performance issues.

Related, we may be able to increase the number of report-creation threads and reduce the time it takes to make the reports (currently 133 minutes), since the initial number of 16 threads predates our AWS infrastructure (we currently use an instance with 8 cores to generate reports).

Note that third-party snapshot creation (which must occur after "regular" snapshot creation) has not moved to threads yet. Those third-party snapshots are created relatively quickly (less than 6 minutes currently), so we won't save all that much time by creating them in parallel. Still, I added a TODO to do that eventually.

🧪 Testing

I tested these changes on my local system using the set of Production organizations, except I neutered the code so that no actual snapshots or reports were generated.

Once I was satisfied with that testing, I created a set of 37 dummy orgs (32 "regular" orgs and 5 "third-party reporting" orgs) and imported them to a test environment (thanks @mcdonnnj for letting me use your environment). I then ran the entire script (well, I commented out a couple of unrelated bits of code that would've required more test environment setup than I was willing to do) and confirmed that the new multithreaded code successfully generated snapshots for the regular orgs and the third-party snapshots were also generated successfully (non-threaded, but using code that was modified in this PR).

✅ Checklist

This PR has an informative and human-readable title.
Changes are limited to a single goal - eschew scope creep!
All future TODOs are captured in issues, which are referenced in code comments.
All relevant type-of-change labels have been added.
I have read the CONTRIBUTING document.
These code changes follow cisagov code standards.
All new and existing tests pass.

… creation

We start with the list of reports to generate, then figure out which snapshots need to be generated in order to create those reports. Finally, we split up the list of snapshots into sublists and launch a snapshot-creation thread to process each sublist. This is similar to how we create reports, though that report code still needs to be cleaned up to match how this snapshot code now works. Note that all snapshots (including snapshots for third-party reports) are now created from a common create_snapshot() function, which simplifies the code slightly.

This is not a huge concern at the moment since third-party snapshot creation does not take very long.

lgtm-com · 2021-01-12T19:25:59Z

This pull request introduces 3 alerts when merging 5bed850 into 6b97eec - view on LGTM.com

new alerts:

2 for Variable defined multiple times
1 for Except block handles 'BaseException'

jsf9k

This looks good to me. I did leave a comment describing an improvement; we may want to create an issue from that.

extras/create_snapshots_reports_scorecard.py

hillaryj

🎉

mcdonnnj

Thanks for getting this task done! 💪💪💪Hopefully it will result in greatly reduced snapshot generation runtime. I had some thoughts/suggestions, but nothing is particularly showstopping.

extras/create_snapshots_reports_scorecard.py

mcdonnnj · 2021-01-14T15:18:30Z

extras/create_snapshots_reports_scorecard.py

+    return snapshot_process.returncode
+
+
+def create_snapshots(org_list, db, cyhy_db_section):


I'm not a fan of create_snapshot()/create_snapshots() because of their similarity. Would something like create_snapshots_from_list() be more at-a-glance readable?

I agree - this would increase usability to disambiguate the two and keep users from having to rely on the help text to tell them apart.

Excellent suggestion- see 36a4128.

extras/create_snapshots_reports_scorecard.py

@mcdonnnj

Thanks @mcdonnnj and LGTM! Co-authored-by: Nick M. <50747025+mcdonnnj@users.noreply.github.com>

lgtm-com · 2021-01-19T15:46:06Z

This pull request introduces 2 alerts when merging 42d9453 into 6b97eec - view on LGTM.com

new alerts:

2 for Variable defined multiple times

Co-authored-by: Nick M. <50747025+mcdonnnj@users.noreply.github.com>

…eate_snapshot()

…inish

dav3r · 2021-01-21T16:45:10Z

I re-requested review from you all to make sure you're all good with the recent changes I made based on your excellent feedback. Thanks!

PS: I re-ran the script in a test environment after making these changes and everything still looks good.

jsf9k

This all still looks great to me! Nice improvements since the last time I reviewed!

hillaryj

Looking good! 🏆

dav3r · 2021-01-22T14:20:41Z

This pull request introduces 2 alerts when merging 42d9453 into 6b97eec - view on LGTM.com

new alerts:

2 for Variable defined multiple times

For posterity, it seems that these LGTM alerts are not taking into account the try ... finally block, as explained in this comment.

mcdonnnj · 2021-01-22T16:07:41Z

This pull request introduces 2 alerts when merging 42d9453 into 6b97eec - view on LGTM.com
new alerts:

2 for Variable defined multiple times

For posterity, it seems that these LGTM alerts are not taking into account the try ... finally block, as explained in this comment.

They are assigned here

cyhy-reports/extras/create_snapshots_reports_scorecard.py

Lines 796 to 797 in 01e36ea

    
           successful_tp_snaps = list() 
        
           failed_tp_snaps = list()

and then if they meet the appropriate logic, they are reassigned here

cyhy-reports/extras/create_snapshots_reports_scorecard.py

Lines 931 to 936 in 01e36ea

    
               successful_tp_snaps = third_party_report_ids 
        
           else: 
        
               # Create snapshots needed for third-party reports 
        
               successful_tp_snaps, failed_tp_snaps = create_third_party_snapshots( 
        
                   db, cyhy_db_section, third_party_report_ids 
        
               )

I think LGTM expects you do define them exclusively in that if/else logic instead of doing an initial declaration and then overriding as needed. I'm fine with how we're doing it, but wanted to give more explanation for posterity.

mcdonnnj

Looking even better to me now! Thanks again for parallelizing this job 💪💪💪

jsf9k · 2021-01-22T16:34:42Z

I think LGTM expects you do define them exclusively in that if/else logic instead of doing an initial declaration and then overriding as needed. I'm fine with how we're doing it, but wanted to give more explanation for posterity.

I detest the leaky scope "feature" of Python.

felddy

👍 Nooice!

dav3r added 9 commits January 6, 2021 09:31

Fix a typo

a89eb06

Clean up description to satisfy linter

69795cc

Improve report-related variable names

fd426e5

Remove old, unused, commented-out code

70ea90c

Add new variables and thread locks to support multi-threaded snapshot…

038463a

… creation

Clean up some logging, variable names, comments, and add a TODO

d208d94

Add a TODO reminder about creating third-party snapshots in threads

ec9525f

This is not a huge concern at the moment since third-party snapshot creation does not take very long.

Add GitHub issue links for TODOs

5bed850

dav3r added the improvement This issue or pull request will add new or improve existing functionality label Jan 12, 2021

dav3r requested review from felddy, jsf9k, hillaryj and mcdonnnj January 12, 2021 19:19

dav3r self-assigned this Jan 12, 2021

This was referenced Jan 12, 2021

Make weekly report generation code consistent with snapshot code #59

Closed

Create third-party snapshots in threads #60

Closed

jsf9k approved these changes Jan 12, 2021

View reviewed changes

extras/create_snapshots_reports_scorecard.py Show resolved Hide resolved

extras/create_snapshots_reports_scorecard.py Outdated Show resolved Hide resolved

hillaryj mentioned this pull request Jan 13, 2021

Determine system requirements for serverless CyHy cisagov/cyhy-system#7

Open

hillaryj approved these changes Jan 13, 2021

View reviewed changes

mcdonnnj approved these changes Jan 14, 2021

View reviewed changes

Avoid catching BaseException

42d9453

Thanks @mcdonnnj and LGTM! Co-authored-by: Nick M. <50747025+mcdonnnj@users.noreply.github.com>

dav3r and others added 5 commits January 21, 2021 10:50

Standardize logging statements for snapshot and report summaries

288be69

Simply logic for creating snapshot command

4024191

Co-authored-by: Nick M. <50747025+mcdonnnj@users.noreply.github.com>

Rename create_snapshots() to reduce confusion with similarly-named cr…

36a4128

…eate_snapshot()

Use logging.error() instead of print("Error: ...")

3344974

Log snapshot completion duration immediately after snapshot threads f…

01e36ea

…inish

dav3r requested a review from mcdonnnj January 21, 2021 16:43

dav3r requested review from hillaryj and jsf9k January 21, 2021 16:43

jsf9k approved these changes Jan 21, 2021

View reviewed changes

hillaryj approved these changes Jan 21, 2021

View reviewed changes

mcdonnnj approved these changes Jan 22, 2021

View reviewed changes

felddy approved these changes Jan 22, 2021

View reviewed changes

dav3r merged commit ebaacc7 into develop Jan 22, 2021

dav3r deleted the improvement/parallelize_snapshot_creation branch January 22, 2021 20:23

dav3r mentioned this pull request Jan 26, 2021

Parallelize snapshot creation #58

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize snapshot creation #61

Parallelize snapshot creation #61

dav3r commented Jan 12, 2021

lgtm-com bot commented Jan 12, 2021

jsf9k left a comment

hillaryj left a comment

mcdonnnj left a comment

mcdonnnj Jan 14, 2021

hillaryj Jan 15, 2021

dav3r Jan 21, 2021

lgtm-com bot commented Jan 19, 2021

dav3r commented Jan 21, 2021

jsf9k left a comment

hillaryj left a comment

dav3r commented Jan 22, 2021

mcdonnnj commented Jan 22, 2021

mcdonnnj left a comment

jsf9k commented Jan 22, 2021

felddy left a comment

		return snapshot_process.returncode


		def create_snapshots(org_list, db, cyhy_db_section):

Parallelize snapshot creation #61

Parallelize snapshot creation #61

Conversation

dav3r commented Jan 12, 2021

🗣 Description

💭 Motivation and Context

🧪 Testing

✅ Checklist

lgtm-com bot commented Jan 12, 2021

jsf9k left a comment

Choose a reason for hiding this comment

hillaryj left a comment

Choose a reason for hiding this comment

mcdonnnj left a comment

Choose a reason for hiding this comment

mcdonnnj Jan 14, 2021

Choose a reason for hiding this comment

hillaryj Jan 15, 2021

Choose a reason for hiding this comment

dav3r Jan 21, 2021

Choose a reason for hiding this comment

lgtm-com bot commented Jan 19, 2021

dav3r commented Jan 21, 2021

jsf9k left a comment

Choose a reason for hiding this comment

hillaryj left a comment

Choose a reason for hiding this comment

dav3r commented Jan 22, 2021

mcdonnnj commented Jan 22, 2021

mcdonnnj left a comment

Choose a reason for hiding this comment

jsf9k commented Jan 22, 2021

felddy left a comment

Choose a reason for hiding this comment