Test run on more program ids with new hst pipeline under hst dir #76

juzen2003 · 2023-10-16T20:02:37Z

Replace Test run on program ids of different instruments #71

…inform the user that pipeline stops if pipeline dir exists and we don't proceed with the next pipeline task.

avaible, if there is no description, the target description tag won't show up in the newly created target label.

the YYYY_MM_DD_HH_MM_SS_PATTERN for the cases like 5217 with some files having IRAF-TLM value "2010-02-17T16:49:27".

…pport.py when trying to figure out the filter name.

remove the undefined function call (citation_information_from_pro)

error and keep the scidata variable to None (line 616-619 in product_labels/__init__.py)

…nning HST pipeline tasks - Add requirements.txt for setup

#65) - In product_labels/__init__.py (line 489-494, 505-510), remove ipppssoot that doesn't have trl timetags from info_by_ipppssoot & info_by_basename to bypass the raise error and continue to run the rest of the pipeline.

so that they can be shared across multiple Popen subprocess. - In task_queue_db.py (line 56-78), define the schema of SubprocessList table. - In task_queue_db.py (line 268-304), create functions of operations for SubprocessList table. - In __init__.py (line 157-196), replace actions on SUBPROCESS_LIST with different operations for SubprocessList table to fetch/update/delete all the subprocesses.

previous task if the next task is to be executed at the end of previous task. - Make sure subprocess list of a proposal id is clear when there is an error raised for that specific proposal id.

- Properly remove the subprocess of from db after each task is done.

update-hst-visit, we will wait until all the visits have completed before queueing finalize-hst-bundle - Make sure when we walked through all the downloaded files from MAST in the staging folder and move them over to the bundles folder, we will remove the organize files in the staging folder.

and pipeline_query_hst_products.py

…_init__.py, queue_manager/config.py and queue_manager/task_queue_db.py

descriptions.

juzen2003 · 2023-12-16T00:18:57Z

Fixed some errors caused by validator.

Modifications:

In PRODUCT_LABEL.xml
- Change hst:moving_target_keyword to hst:moving_target_keywords
- Remove attribute nilReason in hst:gain_setting tag (attribute is not allowed)
- Remove attribute unit in hst:plate_scale tag (attribute is not allowed)
- Remove attribute nilReason in hst:spectral_resolution tag (attribute is not allowed)
- Remove attribute nilReason in hst:center_filter_wavelength tag (attribute is not allowed)
- Remove attribute nilReason in hst:bandwidth tag (attribute is not allowed)
- Add unit="byte" to record_length, field_location, and field_location tags
In hst_dictionary_support.py
- Change hst:observation_type from SPECTROSCOPIC back to SPECTROGRAPHIC
- Change Array_1D to Array
In PRODUCT_LABEL.xml
- Change File_Area to File_Area_Ancillary tag if processing level is "Ancillary"
In all labels templates, keep logical_identifier tag and its content in one line to avoid validator complaining about the unexpected carriage returns

Issues that needs to be reviewed/discussed:

hst:gain_setting, hst:spectral_resolution, hst:center_filter_wavelength, and hst:bandwidth values need to match this pattern r'(\+|-)?([0-9]+(\.[0-9]*)?|\.[0-9]+)([Ee](\+|-)?[0-9]+)?|[^aFIN,]* ', what should we put if it's not applicable? Remove the tag? (empty string is not a valid value)
No child element allowed for these tags" hst:visit_id & hst:Processing_Parameters
For hst:observation_type, it must exists, and the validator requires the value to be one of these: 'IMAGING', 'SPECTROGRAPHIC', 'TIME-SERIES', so 'UNK' will fail the validator.
For data class, these are the available values: {Array, Array_2D, Array_2D_Image, Array_2D_Map, Array_2D_Spectrum, Array_3D, Array_3D_Image, Array_3D_Movie, Array_3D_Spectrum, Checksum_Manifest, Encoded_Header, Encoded_Image, Header, Stream_Text, Table_Binary, Table_Character, Table_Delimited}. Should we modify DATA_CLASS_TO_NOUN in hdu_data_descriptions.py? What value should we put for Array_1D_Spectrum?

all nilReason attributes.

that doesn't exist

juzen2003 · 2024-01-22T19:19:02Z

Current changes: (updates after 1/19/24 meeting)

In PRODUCT_LABEL.xml
- Change hst:moving_target_keyword to hst:moving_target_keywords
- Add unit="byte" to record_length, field_location, and field_location tags
In hst_dictionary_support.py
- Change hst:observation_type value SPECTROSCOPIC to SPECTROGRAPHIC
- Change File_Area to File_Area_Ancillary tag if processing level is "Ancillary"
In all labels templates, keep logical_identifier tag and its content in one line to avoid validator complaining about the unexpected carriage returns
All other changes made in previous comment except the above listed ones are reverted.

Pending items:

Get HST dicitonary definition update

….xml

- Remove unused SCHEMAS & SCHEMATRONS

- Remove unit attribute in the hst:plate_scale tag, will put it back later.

returned time format.

subprocess is removed.

label_hst_context_directory function in finalize_context.py

Target tag if the target has no description.

folders.

change it back to 'Array_1D_Spectrum' once DDWG has an update.

<Observing_System> in the bundle, product collection, and product label templates.

tag content under <Identification_Area> tag to <Context_Area>.<comment>

will have value_data_type for hst:moving_target_description changed from ASCII_Short_String_Collapsed to ASCII_String

juzen2003 · 2024-06-17T19:09:26Z

Details of the new updates in #56 (comments on 06/17/24)

matthewtiscareno · 2024-06-18T01:45:36Z

Details of the new updates in #56 (comments on 06/17/24)

Yes, those look good. So do all the tests pass now?

for context and document collection.

of parsing the name tag using the removed get_clean_target_text function

files for different proposal ids. This will avoid failure caused by multiple bundles trying to update tmp-context-products.json at the same time when running the validator.

juzen2003 · 2024-06-21T17:18:17Z

06/21/24 updates:
When creating the target label that doesn't exist in https://pds.nasa.gov/data/pds4/context-pds4/target/ , obtain the correct type and name from lid for the target label.

juzen2003 added 17 commits July 6, 2023 12:12

Update the logger message on pipeline_query_hst_moving_targets.py to …

f3b733f

…inform the user that pipeline stops if pipeline dir exists and we don't proceed with the next pipeline task.

Update YEARS_FOR_CYCLE in citations/__init__.py to include (2022, 2023)

9e05351

Update TARGET_LABEL.xml to include target description tag only when it's

b6f6eb2

avaible, if there is no description, the target description tag won't show up in the newly created target label.

Update line 128-129 in data_support.py to make sure that we also check

9e20ca0

the YYYY_MM_DD_HH_MM_SS_PATTERN for the cases like 5217 with some files having IRAF-TLM value "2010-02-17T16:49:27".

Fixed false return of hst_dictionary at line 548 in hst_dictionary_su…

1593ccd

…pport.py when trying to figure out the filter name.

Removed unused print statement at line 1000 in hst_dictionary_support.py

94bdded

Update to make sure we get the citations info from .apt and .pro, also

7ea5bcc

remove the undefined function call (citation_information_from_pro)

Update the code to handle time out error when trying to access MAST.

c0a9962

When there is no 'SCIDATA' in the spt_hdulist header, we bypass the key

7b26065

error and keep the scidata variable to None (line 616-619 in product_labels/__init__.py)

- Add setup.md under HST/doc to describe the required setup before ru…

2e32a46

…nning HST pipeline tasks - Add requirements.txt for setup

- Update wait_for_subprocess to open the subprocess slot occupied by

f89130a

previous task if the next task is to be executed at the end of previous task. - Make sure subprocess list of a proposal id is clear when there is an error raised for that specific proposal id.

- Update the intelligence of wait_for_subprocess

480a979

- Properly remove the subprocess of from db after each task is done.

Removed all commented out code in pipeline_query_hst_moving_targets.py

c00329a

and pipeline_query_hst_products.py

Update requirements.txt

d6fb6af

juzen2003 mentioned this pull request Oct 16, 2023

Test run on program ids of different instruments #71

Closed

juzen2003 added 7 commits October 19, 2023 14:34

Update & clean up the code based on Rob's feedback in queue_manager/_…

4ff0a33

…_init__.py, queue_manager/config.py and queue_manager/task_queue_db.py

Change all 'mast' and 'Mast' to 'MAST' for files under HST/

b756016

Re-org files, move the old files under archive/

2c7e4ef

Update requirements.txt to work with python 3.11

6268ee3

Add example commands on how to run the program in setup document.

237b1c8

Add title Example commands for commands section.

cdac036

Clean up a logger message at line 155 of queue_manager/__init__.py

63b2fc9

rfrenchseti requested review from markshowalter and rfrenchseti October 23, 2023 18:10

juzen2003 added 3 commits October 26, 2023 23:52

Remove the ending periods in the logger messages.

a2768cc

Turn off sqlalchemy console messages.

fb98bed

Update all pipeline programs to have the correct filenames in the parser

f68d425

descriptions.

juzen2003 mentioned this pull request Dec 16, 2023

Product label errors flagged by validation tool #56

Open

juzen2003 added 3 commits January 19, 2024 12:27

Restore data class to 'Array_1D'

629846b

Restore removed nilReason attributes and add xsi:nil="true" in front of

5a1d94c

all nilReason attributes.

Add exception to handle errors raised when trying to remove a folder

631a7d9

that doesn't exist

juzen2003 added 15 commits January 24, 2024 10:50

Change the IM version from 1D to 1L in xml/PDS4_HST_1D00_1000-new.xsd…

e94235e

….xml

Update requirements.txt to work with python 3.12

8d6c8e3

- Change moving_target_keywords to moving_target_keyword

4cc439e

- Remove unused SCHEMAS & SCHEMATRONS

Remove commented out code in validate-pdart

f0e8c8d

- Make sure collection type is correct in all collection labels.

facb0ec

- Remove unit attribute in the hst:plate_scale tag, will put it back later.

Update get_time_coordinates function to append 'Z' at the end of the

0b09bca

returned time format.

Change 'SPECTROGRAPHIC' to 'SPECTROSCOPIC' in hst_dictionary_support.py

a902a8f

Update wait_for_subprocess function by improving the log messages when a

5be3364

subprocess is removed.

Fixed the calculation of records_num for context collection by updating

b9f5613

label_hst_context_directory function in finalize_context.py

Update TARGET_LABEL.xml to add 'none' value to the description tag under

bad0fdf

Target tag if the target has no description.

Avoid removing non existing staging directories when cleaning up staging

8da795d

folders.

Use 'Array_1D' for hdu data class 'Array_1D_Spectrum' for now, will

43a6425

change it back to 'Array_1D_Spectrum' once DDWG has an update.

Fixed the contents of <Observing_System_Component> tag under

52475fd

<Observing_System> in the bundle, product collection, and product label templates.

Update product label template, move the "note that" part of the <title>

7e348c3

tag content under <Identification_Area> tag to <Context_Area>.<comment>

Update to the latest schemas and schematrons for validator. This one

3220d5e

will have value_data_type for hst:moving_target_description changed from ASCII_Short_String_Collapsed to ASCII_String

juzen2003 added 4 commits June 17, 2024 21:15

Update the test results for craeting labels. Update the golden labels

90dd2ed

for context and document collection.

Update get_clean_target_text to remove '/' in the target name.

c5bd9e8

Obtain the target identification name and type from the lid tag instead

b599fa8

of parsing the name tag using the removed get_clean_target_text function

Update validation process to use different tmp-context-products json

a831da5

files for different proposal ids. This will avoid failure caused by multiple bundles trying to update tmp-context-products.json at the same time when running the validator.

juzen2003 merged commit 96b683e into main Jun 21, 2024

juzen2003 deleted the test_run_on_more_program_ids_with_new_hst_pipeline_under_HST_dir branch June 21, 2024 17:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test run on more program ids with new hst pipeline under hst dir #76

Test run on more program ids with new hst pipeline under hst dir #76

juzen2003 commented Oct 16, 2023

juzen2003 commented Dec 16, 2023

juzen2003 commented Jan 22, 2024 •

edited

Loading

juzen2003 commented Jun 17, 2024

matthewtiscareno commented Jun 18, 2024

juzen2003 commented Jun 21, 2024

Test run on more program ids with new hst pipeline under hst dir #76

Test run on more program ids with new hst pipeline under hst dir #76

Conversation

juzen2003 commented Oct 16, 2023

juzen2003 commented Dec 16, 2023

juzen2003 commented Jan 22, 2024 • edited Loading

juzen2003 commented Jun 17, 2024

matthewtiscareno commented Jun 18, 2024

juzen2003 commented Jun 21, 2024

juzen2003 commented Jan 22, 2024 •

edited

Loading