Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create archive page asynchronously #1203

Conversation

bhilbert4
Copy link
Collaborator

Rather than querying the database and constructing the context for an instrument's archive page at the time of the request, do that work asynchronously, and save the results in a json file. Then when there is a request for the page, read in the json file and create the page.

@bhilbert4 bhilbert4 self-assigned this Feb 28, 2023
@bhilbert4 bhilbert4 linked an issue Feb 28, 2023 that may be closed by this pull request
@bhilbert4
Copy link
Collaborator Author

Github won't let me link it, but this should also resolve #959

@bhilbert4
Copy link
Collaborator Author

For this to work on the servers, we need to create the os.path.join(get_config('outputs'), "archive_page") directory.

@bhilbert4
Copy link
Collaborator Author

On the test server:
Time needed to load the nircam archive page using the develop branch: 1 minute and 45 seconds
Time needed when using this branch: 13 seconds.
🥳

@bhilbert4
Copy link
Collaborator Author

@mfixstsci I think this is ready for review.

@bhilbert4 bhilbert4 changed the title [WIP]: Create archive page asynchronously Create archive page asynchronously Feb 28, 2023
@bhilbert4
Copy link
Collaborator Author

With these changes, the json files containing the content of the archive pages are updated after each run of archive_database_update.py, which is done via cron job.

@bhilbert4 bhilbert4 linked an issue Feb 28, 2023 that may be closed by this pull request
@bhilbert4
Copy link
Collaborator Author

In addition to creating the new archive_page subdirectory under the outputs directory, we'll also need to run the new script on all of the instruments when we merge this.

from datetime import datetime
from jwql.website.apps.jwql.data_containers import create_archived_proposals_context
insts = ['nircam', 'miri', 'nirspec', 'niriss', 'fgs']
for inst in insts:
   print(f'Working on {inst}, {datetime.now()}')
   create_archived_proposals_context(inst)

Copy link
Collaborator

@mfixstsci mfixstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @bhilbert4 I am going to rebase the branch and then merge it once the CI is done.

# Read in the json file created by data_containers.create_archived_proposals_context
# and use as the context
output_dir = get_config()['outputs']
context_file = os.path.join(output_dir, 'archive_page', f'{inst}_archive_context.json')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slick 👍

@mfixstsci mfixstsci merged commit b9794cc into spacetelescope:develop Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Archive page for NIRCam is very slow to load Speed up getting the number of files for a proposal
2 participants