-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make sure there are not duplicate TROPOMI files #125
Conversation
src/utilities/test_TROPOMI_dir.py
Outdated
number_of_unique_orbit_numbers = len(set(all_orbit_numbers)) # forming a set drops the duplicates | ||
number_of_files_in_Sat_datadir = len(files) | ||
|
||
assert number_of_unique_orbit_numbers == number_of_files_in_Sat_datadir, "There are duplicate TROPOMI datafiles (as defined by their orbit number)." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe update this to give more guidance? Eg. Error: Duplicate orbit numbers found in TROPOMI datafiles. Remove duplicate files from system before continuing.
Also, does assert trigger a non-zero exit code? Does the IMI exit when it hits this error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what you mean by a non-zero exit code. The IMI does exit when it hits this error.
=== PARSING CONFIG FILE (run_imi.sh) ===
Activating conda environment: imi_env
mkdir: created directory ‘/n/holyscratch01/jacob_lab/nbalasus/Test_IMI’
Traceback (most recent call last):
File "/n/home06/nbalasus/IMI_test_tropomi_duplicates/integrated_methane_inversion/src/utilities/test_TROPOMI_dir.py", line 27, in <module>
check_for_duplicate_orbit_numbers(Sat_datadir)
File "/n/home06/nbalasus/IMI_test_tropomi_duplicates/integrated_methane_inversion/src/utilities/test_TROPOMI_dir.py", line 23, in check_for_duplicate_orbit_numbers
assert number_of_unique_orbit_numbers == number_of_files_in_Sat_datadir, "Duplicate orbit numbers found in TROPOMI datafiles. Remove duplicate files from system before continuing."
AssertionError: Duplicate orbit numbers found in TROPOMI datafiles. Remove duplicate files from system before continuing.
FATAL ERROR on line number 126 of slurm_script: IMI exiting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay perfect, thanks.
FYI It is standard practice when running a program to exit(0)
upon successful completion. Non-zero exit codes denote an error. We have a line in run_imi.sh (trap 'imi_failed $LINENO' ERR
) that catches any non-zero exit codes and calls a standard error message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
We had an issue in the past where
download_TROPOMI.py
would result in users having duplicate TROPOMI files (e.g., two files with orbit number xxxxx, but one with processor version 02.04.00 and one with processor version 01.03.00). This simple and fast test should prevent this in the future as processor version numbers continue to change.