Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature #2283 include times #2345

Merged
merged 94 commits into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
3c74be4
added helper function to clean up logic to add new dictionaries that …
georgemccabe Aug 8, 2023
60662e4
fix incorrect variable name
georgemccabe Aug 8, 2023
927d631
log error if wrapper was not loaded properly to help debug issue
georgemccabe Aug 8, 2023
ffe1f23
improve logging to print input dir/template once at beginning to avoi…
georgemccabe Aug 8, 2023
fee61a2
make wrappers LoopTimes (RuntimeFreq using run all times) to remove r…
georgemccabe Aug 8, 2023
89a34d8
fix function name in test, ci-run-all-diff
georgemccabe Aug 9, 2023
a3f6c70
set self.app_name if unset for all wrappers to prevent errors when cr…
georgemccabe Aug 9, 2023
5673a61
set self.app_name if unset for all wrappers to prevent errors when cr…
georgemccabe Aug 9, 2023
d82c5e8
change all CompareGridded (GridStat, PointStat, EnsembleStat, MODE, M…
georgemccabe Aug 9, 2023
39e8582
add test for usage wrapper, ci-run-all-diff
georgemccabe Aug 9, 2023
6dbc6aa
fix call to RegridDataPlane from ExtractTiles wrapper
georgemccabe Aug 9, 2023
82f04e4
changed more wrappers to use consistent time looping, ci-run-all-cases
georgemccabe Aug 9, 2023
469bbe5
formatting
georgemccabe Aug 9, 2023
83e0286
more formatting
georgemccabe Aug 9, 2023
9cb8b7e
remove unnessary check for skip times because it is called upstream, …
georgemccabe Aug 9, 2023
da19788
add SKIP_LEAD_SEQ check to RuntimeFreq run_at_time and change TCPairs…
georgemccabe Aug 9, 2023
f19aaf5
run each time input through ti_calculate before passing it to run_at_…
georgemccabe Aug 10, 2023
e0a68d2
move runtime log banner
georgemccabe Aug 11, 2023
21f8f27
rename function to avoid confusion with other find_input_files functions
georgemccabe Aug 11, 2023
ba81987
refactored logic in ti_calculate and added unit tests for common inpu…
georgemccabe Aug 14, 2023
3bc665a
formatting cleanup
georgemccabe Aug 14, 2023
92d5d26
moved logic to compute formatted init or valid so it is still run eve…
georgemccabe Aug 14, 2023
2c60cd4
pass data type to find_data and change template c_dict key if data ty…
georgemccabe Aug 14, 2023
77987c2
refactor MTD wrapper to use RuntimeFreq methods to find files to proc…
georgemccabe Aug 14, 2023
a1ba59b
use f-strings for command, ci-run-all-diff
georgemccabe Aug 14, 2023
c9610e0
added unit test for case that happens in gfdl tracker wrapper
georgemccabe Aug 14, 2023
95a245f
formatting
georgemccabe Aug 14, 2023
55dbcd8
fixed failures when trying to format times that are wildcards, fix ch…
georgemccabe Aug 14, 2023
c60b3eb
set default runtime freq for TCPairs wrapper
georgemccabe Aug 14, 2023
9bac908
fix use case to use init instead of valid, ci-run-all-diff
georgemccabe Aug 14, 2023
460851e
Merge branch 'develop' into feature_2283_include_times
georgemccabe Aug 15, 2023
9b7ee25
preserve old behavior by using init and first lead to compute valid t…
georgemccabe Aug 15, 2023
c133bb4
if [INIT/VALID]_BEG == [INIT/VALID]_END and runtime freq is RUN_ONCE,…
georgemccabe Aug 15, 2023
70fa3cb
fix bug introduced by using RuntimeFreq that causes duplicate regex g…
georgemccabe Aug 15, 2023
9a433e2
fix config to use same init beg/end for RUN_ONCE mode
georgemccabe Aug 15, 2023
4c903c6
use RuntimeFreq time looping, preserve old behavior by setting runtim…
georgemccabe Aug 15, 2023
95eac30
turn on use case groups that were failing to test , ci-run-diff
georgemccabe Aug 15, 2023
334f921
add missing import, ci-run-diff
georgemccabe Aug 15, 2023
9e1eeab
name file_list file with 'single' if MTD_SINGLE_RUN to prevent diffs …
georgemccabe Aug 15, 2023
0d53a0c
remove unused variable
georgemccabe Aug 15, 2023
101f5d0
call self.clear before running in RUN_ONCE mode to clear out command …
georgemccabe Aug 15, 2023
1e4a953
change TCGen wrapper to RuntimeFreq
georgemccabe Aug 15, 2023
df2d18a
set lead_hours/minutes/seconds even if lead is months or years so rel…
georgemccabe Aug 15, 2023
9a1a610
update use case config to use single run time (RUN_ONCE) and set init…
georgemccabe Aug 15, 2023
569c6cf
turn off use cases that now pass and test if other cases will pass, c…
georgemccabe Aug 15, 2023
50fbe5a
turn off all use cases
georgemccabe Aug 15, 2023
4383869
Create class variables for RuntimeFreq wrappers to define default val…
georgemccabe Aug 15, 2023
2bcda33
error and return if start and end times cannot be read, e.g. if end t…
georgemccabe Aug 15, 2023
8f308ff
update config files to use current config options, ci-run-all-diff
georgemccabe Aug 15, 2023
74e52bd
fixed bug when beg/end times are not set and aren't needed, ci-run-a…
georgemccabe Aug 15, 2023
b9eef8f
fix use case reference to init time
georgemccabe Aug 16, 2023
75da7ab
update contrib guide
georgemccabe Aug 16, 2023
407aba4
clean up contrib guide additions
georgemccabe Aug 16, 2023
cef425d
change supported runtime frequency values for wrappers that need enha…
georgemccabe Aug 16, 2023
4ed89d6
fix job args to remove hours from -init_beg and -init_end
georgemccabe Aug 16, 2023
fea1c43
per #2283, add support for INCLUDE_TIMES (opposite of SKIP_TIMES) and…
georgemccabe Aug 16, 2023
9739279
Merge branch 'feature_2283_time_looping' into feature_2283_include_times
georgemccabe Aug 16, 2023
c5a538e
move timing log output of group so it can be easily viewed without op…
georgemccabe Aug 16, 2023
6ef7f6d
per #2283, add documentation about INCLUDE_TIMES and using days of th…
georgemccabe Aug 16, 2023
58fcc2f
create URL to easily navigate to commit that triggered workflow run
georgemccabe Aug 29, 2023
a90e31b
turn on a use case that is failing due to this issue
georgemccabe Sep 5, 2023
5dc6222
updated version of Docker base image used for all conda env images to…
georgemccabe Sep 5, 2023
ab32aca
added instructions to rebuild conda.vX.Y images as needed
georgemccabe Sep 5, 2023
21c31c1
change version extension of new docker images for conda environments …
georgemccabe Sep 6, 2023
6a1ec46
try building geovista via conda to see if required functionality is a…
georgemccabe Sep 6, 2023
a62feab
per #2283, implement skip and include times for init and valid
georgemccabe Sep 6, 2023
66f69c5
merged develop and resolved conflicts
georgemccabe Sep 6, 2023
cb33318
per #2283, add unit tests to ensure skip/include init/valid times wor…
georgemccabe Sep 6, 2023
9b18228
add other packages that were missing
georgemccabe Sep 6, 2023
6491808
add files to create conda envs via GHA workflow
georgemccabe Sep 8, 2023
9e6637b
use job ID to set env var used to name conda environment -- note cann…
georgemccabe Sep 8, 2023
acbebce
whitespace
georgemccabe Sep 8, 2023
07d7278
undo whitespace change to, ci-run-all-diff
georgemccabe Sep 8, 2023
0a10dff
renamed function that was changed in geovista between v0.1.4 and v0.2.0
georgemccabe Sep 8, 2023
25dd1f1
remove specific versions from packages
georgemccabe Sep 11, 2023
95489f8
remove versions from packages
georgemccabe Sep 11, 2023
205fb88
removed unused environment
georgemccabe Sep 11, 2023
6c85d5f
update use case to replace use of deprecated pandas DataFrame.append …
georgemccabe Sep 11, 2023
273287f
update geovista env to install libGL and libEGL to prevent dynamic li…
georgemccabe Sep 12, 2023
7af0704
update steps to install Java OpenJDK for GempakToCF.jar
georgemccabe Sep 12, 2023
7d1cdf6
copy files from new java JDK location
georgemccabe Sep 12, 2023
f90c3e4
run use case that is failing due to gempak install issues
georgemccabe Sep 12, 2023
061317d
copy additional java files needed to run
georgemccabe Sep 12, 2023
2803c92
turn off use case now that it passes
georgemccabe Sep 12, 2023
2a64e07
added version numbers for python packages to ensure the env is reprod…
georgemccabe Sep 12, 2023
56262e2
Update internal/scripts/docker_env/Dockerfile.conda
georgemccabe Sep 12, 2023
5a09634
fixed incorrect info in input description
georgemccabe Sep 12, 2023
2fead00
Merge branch 'feature_2338_debian12_conda_envs' of github.com:dtcente…
georgemccabe Sep 12, 2023
787e037
merged feature branch that is in an open PR
georgemccabe Sep 13, 2023
63a3399
clarify version numbers
georgemccabe Sep 13, 2023
0da2c3f
add new variables to glossary/wrappers for each wrapper-specific vari…
georgemccabe Sep 13, 2023
4cba461
fix doc errors
georgemccabe Sep 13, 2023
b3569ed
fix function documentation
georgemccabe Sep 13, 2023
8ce5375
Merge branch 'develop' into feature_2283_include_times
georgemccabe Sep 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/jobs/docker_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,13 @@ def run_commands(commands):
error_message = f"ERROR: Command failed -- {err}"
is_ok = False

print("::endgroup::")

end_time = time.time()
print("TIMING: Command took "
f"{time.strftime('%M:%S', time.gmtime(end_time - start_time))}"
f" (MM:SS): '{command}')")

print("::endgroup::")

if error_message:
print(error_message)

Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ jobs:
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
run: echo "$GITHUB_CONTEXT"
- name: Build URL for commit that triggered workflow
if: github.event_name == 'workflow_dispatch'
run: echo https://github.com/${{ github.event.inputs.repository }}/commit/${{ github.event.inputs.sha }}

job_control:
name: Determine which jobs to run
Expand Down
Loading