Build archive page from database

[bhilbert4]

Resolves #1023

This PR changes the way an instrument's archive page is built. The task of querying MAST for each proposal and then working through all of the files for that proposal has been moved to a script that can be run any time. The results of the script are used to populate new database tables.

Then when someone navigates to an instrument's archive page, the database table is loaded and the information is used to build the page.

[pep8speaks]

Hello @bhilbert4, Thank you for updating !

• In the file jwql/website/apps/jwql/archive_database_update.py:

Line 39:1: E402 module level import not at top of file
Line 40:1: E402 module level import not at top of file
Line 41:1: E402 module level import not at top of file
Line 42:1: E402 module level import not at top of file
Line 43:1: E402 module level import not at top of file
Line 44:1: E402 module level import not at top of file
Line 45:1: E402 module level import not at top of file
Line 49:1: E302 expected 2 blank lines, found 1
Line 209:5: E265 block comment should start with '# '
Line 209:5: E303 too many blank lines (3)
Line 210:5: E265 block comment should start with '# '
Line 211:5: E265 block comment should start with '# '
Line 213:5: E265 block comment should start with '# '
Line 217:5: E303 too many blank lines (3)
Line 219:5: E265 block comment should start with '# '
Line 234:5: E265 block comment should start with '# '
Line 234:5: E303 too many blank lines (2)
Line 237:5: E265 block comment should start with '# '
Line 244:5: E303 too many blank lines (3)
Line 250:9: E265 block comment should start with '# '
Line 251:9: E265 block comment should start with '# '
Line 262:17: E265 block comment should start with '# '
Line 265:17: E265 block comment should start with '# '
Line 266:17: E265 block comment should start with '# '
Line 283:9: E265 block comment should start with '# '
Line 289:9: E265 block comment should start with '# '
Line 290:9: E265 block comment should start with '# '
Line 323:9: E265 block comment should start with '# '
Line 326:9: E265 block comment should start with '# '
Line 332:9: E265 block comment should start with '# '
Line 340:1: E303 too many blank lines (3)

• In the file jwql/website/apps/jwql/views.py:

Line 275:5: E265 block comment should start with '# '
Line 279:5: E265 block comment should start with '# '

Comment last updated at 2022-09-13 17:59:11 UTC

[bhilbert4]

This is still very much a work in progress, but I figured I should get the skeleton of it uploaded. I'm always very slow when trying to work with databases.

[BradleySappington]

Initial thoughts that we will eventually implement an "Image" class to hold the observation specific thumbnails and filterable data. This Image class should be where we have the Exposure Type kept. Exposure Types can just be defined and stored as objects in the Image class (along with name and thumbnail location).

[BradleySappington]

can also decide if we want to handle images and thumbnails using the FileField, ImageField, or just maintaining a path location as is

[bhilbert4]

The new page is working, but is still slower than I had anticipated.

Looking at how long it's taking to do various things, the database access appears fairly fast. The slowest task is generating the list of proposal numbers. Timing results are below for the updated archived_proposals_ajax. Values are in units of seconds. Each "proposal loop" iteration shows the results when working on one proposal. Everything within the proposal loop is small (5-20ms), compared to getting the list of proposals (10-12 seconds). And note that this is timing the tasks in archived_proposals_ajax only. This doesn't say anything about archived_proposals nor the JS. But still, this argues for finding a faster way to get a list of proposals.

Initial filter: 0.0008904417045414448
Get list of proposals: 12.152243920136243
In proposal loop:
filter: 0.00029786117374897003
thumbnail entry: 0.02204055292531848
get obsnums: 0.008105554152280092
get filecount: 5.180947482585907e-06

In proposal loop:
filter: 0.0003039059229195118
thumbnail entry: 0.022973330225795507
get obsnums: 0.014654692728072405
get filecount: 6.702262908220291e-06

In proposal loop:
filter: 0.00036222394555807114
thumbnail entry: 0.015787335112690926
get obsnums: 0.007101284805685282
get filecount: 5.035195499658585e-06

[bhilbert4]

@mfixstsci this is ready for review. Works well when tested on the test server.

[bhilbert4]

Ah , right. I forgot about PEP8...

[bhilbert4]

Hmm, PEP8 seems to be stuck checking an old version of the files.

[mfixstsci]

Looks great, I really like this models based approach. thanks for submitting this!

[mfixstsci]

I also checked out the pep8 issues listed here and pulled your repo locally and it looked fine 🤷‍♂️