Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove duplicate filenames from tables #45

Merged
merged 3 commits into from
Apr 21, 2023

Conversation

bhilbert4
Copy link
Collaborator

Resolves #26

In some cases, multiple copies of a given filename are present in the tables created from the MAST queries. This leads to repeated downloads of the same file. This PR remove the duplicate entires.

@bhilbert4 bhilbert4 self-assigned this Apr 14, 2023
@bhilbert4
Copy link
Collaborator Author

Duplicates have been removed, but the product search does not make use of the starting and stopping time limits, so it is returning all files for the given propID.

  • Need to find a way to associate start times with the rows in the product table, and then filter out entries that are outside of the user-given datetime bounaries.

@bhilbert4
Copy link
Collaborator Author

bhilbert4 commented Apr 17, 2023

Looks like maybe we need to match up the parent_obsid from the products query with the obsid from the obs query. Then we can apply the t_min value from the obs query to the individual products from the product query, and filter appropriately.

Hmmm, that actually won't work. In the obsTable, all we have is the start time for the observation, not for individual files or even visits. So if the user requests an end time that is within the observation, we can't use the observation t_min to find the matching files.

@bhilbert4
Copy link
Collaborator Author

bhilbert4 commented Apr 17, 2023

Tested the updated code using the example in #26. The number of files returned now matches what I find manually. Duplicates have been removed. Rather than trying to download a list of 27730 files, it now has a list of 158 uncal files plus 8 msa files.

@bhilbert4 bhilbert4 changed the title [WIP]: Remove duplicate filenames from tables Remove duplicate filenames from tables Apr 17, 2023
@bhilbert4
Copy link
Collaborator Author

Merging this now, as it solves the problem of downloading multiple copies of files. Speeding up the query or making it smarter will come in a future PR.

@bhilbert4 bhilbert4 merged commit 188f18a into spacetelescope:main Apr 21, 2023
@bhilbert4 bhilbert4 deleted the repeated-files branch April 21, 2023 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

File number Issues with NIRSpec MSA downloads
1 participant