Define output dict for multi species in main.py #735

JintaoWu98 · 2024-02-14T12:43:32Z

We have problem when we launch restart.yml saying unexpected keyword argument 'output_multi_spc'. It turns out previously we defined Scheduler.output_multi_spc parallel to Scheduler.output in Scheduler __init__, however, we forgot to define it in the ARC class in main.py. So now we add the relevant terms in main.py.

codecov · 2024-02-14T13:34:24Z

Codecov Report

Attention: Patch coverage is 28.57143% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 73.81%. Comparing base (be1f6c8) to head (5ca8ae1).

Files	Patch %	Lines
arc/scheduler.py	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #735      +/-   ##
==========================================
- Coverage   73.82%   73.81%   -0.01%     
==========================================
  Files          99       99              
  Lines       27346    27352       +6     
  Branches     5717     5718       +1     
==========================================
+ Hits        20187    20189       +2     
- Misses       5733     5737       +4     
  Partials     1426     1426

Flag	Coverage Δ
unittests	`73.81% <28.57%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

alongd

Thanks! Can you please see the two comments below?

alongd · 2024-02-14T18:56:39Z

arc/main.py

@@ -180,6 +180,8 @@ class ARC(object):
            Job types not defined in adaptive levels will have non-adaptive (regular) levels.
        output (dict): Output dictionary with status and final QM file paths for all species. Only used for restarting,
                       the actual object used is in the Scheduler class.
+        output_multi_spc (dict): Output dictionary with status and final QM file paths for the multi species. 


please also add above under Args:

alongd · 2024-02-14T18:57:30Z

arc/main.py

@@ -291,6 +294,7 @@ def __init__(self,
        if not os.path.exists(self.project_directory):
            os.makedirs(self.project_directory)
        self.output = output
+        self.output_multi_spc = output_multi_spc


we store it as an attribute, but we don't do anything with it. I think we should transmit it to Scheduler (and also add it as an argument in Scheduler like here)

We actually transmitted it to Scheduler as an attribute in the previous PR already (like what had been done to the output dict), I think we just forgot to define it in the main.py.

I could be misunderstanding but I think @alongd meant here

ARC/arc/main.py

Line 591 in 688f960

self.scheduler = Scheduler(project=self.project,

Also, do we need it part of this function for the dumping of the yaml? @alongd
https://github.com/ReactionMechanismGenerator/ARC/blob/688f9602e7d41252cc0e22a27cd30a4d59eafcf5/arc/main.py#L464C1-L467C12

I could be misunderstanding but I think @alongd meant here

ARC/arc/main.py

Line 591 in 688f960

self.scheduler = Scheduler(project=self.project,

Thank you for pointing that out. Indeed, we didn't transmit the output dictionary to the Scheduler. I have a question, though. I thought output_multi_spc was the multi-species counterpart to output. However, since we didn't transmit output here, why is there a need to transmit output_multi_spc? Am I missing something?

Ah maybe you're right. I'm not really familiar with the restart function. Do you know with these changes you've made that if a restart of a multi species works? As in ARC picks up that a multi species is/was run(ning)

I'm not that familiar with the restart function either. Currently, I only have restart.yml from completed projects (both single and multi-species), which have already finished running. With these changes, I launched restart.yml for single and multi-species projects individually and compared them. They do yield similar folders, such as log_and_restart_archive, and no errors were reported in the err.txt. However, I'm not sure how this will turn out in the future with an unfinished project.

Areyou able to do with it unfinished. What I mean is start the job with the input.yml and then as soon as you see it submits its first job to the server, cancel ARC qdel <job id> and then restart it

Now I get the following error, this is the error when I terminate the ARC submitted job

Traceback (most recent call last): File "/home/jintaowu/Code/ARC/ARC.py", line 69, in <module> main() File "/home/jintaowu/Code/ARC/ARC.py", line 65, in main arc_object.execute() File "/home/jintaowu/Code/ARC/arc/main.py", line 620, in execute skip_nmd=self.skip_nmd, File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 508, in __init__ self.schedule_jobs() File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 595, in schedule_jobs successful_server_termination = self.end_job(job=job, label=label, job_name=job_name) File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 993, in end_job self._run_a_job(job=job, label=label) File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 1060, in _run_a_job xyz=job.xyz, File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 801, in run_job checkfile = self.species_dict[label].checkfile if isinstance(label, str) else None KeyError: 'multi_spc1'

And this is the error when I restart the job

Traceback (most recent call last): File "/home/jintaowu/Code/ARC/ARC.py", line 69, in <module> main() File "/home/jintaowu/Code/ARC/ARC.py", line 65, in main arc_object.execute() File "/home/jintaowu/Code/ARC/arc/main.py", line 620, in execute skip_nmd=self.skip_nmd, File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 483, in __init__ self.run_opt_job(species.label, fine=self.fine_only) File "/home/jintaowu/Code/ARC/arc/scheduler.py", line 1188, in run_opt_job if self.output_multi_spc[self.species_dict[label].multi_species].get(key, False): KeyError: 'multi_spc1'

arc/main.py

arc/scheduler.py

main.py adds `output_multi_spc` as key in ARC.as_dict make output_multi_spc in restart_dict not None

alongd · 2024-04-16T05:44:45Z

I confirm that ARC's restart does not crash on this branch. @calvinp0, do you have any additional comments, or can we merge?

calvinp0 · 2024-04-16T18:49:56Z

I confirm that ARC's restart does not crash on this branch. @calvinp0, do you have any additional comments, or can we merge?

The code scanning issues need to be resolved I think

JintaoWu98 · 2024-04-16T20:49:47Z

I confirm that ARC's restart does not crash on this branch. @calvinp0, do you have any additional comments, or can we merge?

The code scanning issues need to be resolved I think

@calvinp0, thanks for pointing it out.

@alongd, the updated branch is now more straightforward and robust.

If there are any other comments, please feel free to let me know.

calvinp0

Thanks @JintaoWu98!

github-actions bot added the Module: Main label Feb 14, 2024

JintaoWu98 requested review from alongd and calvinp0 February 14, 2024 12:43

alongd reviewed Feb 14, 2024

View reviewed changes

JintaoWu98 force-pushed the restart_multi_spc branch 3 times, most recently from f4f0dd9 to 5343194 Compare February 16, 2024 10:08

JintaoWu98 force-pushed the restart_multi_spc branch from 5343194 to a21c066 Compare March 3, 2024 13:57

github-actions bot added Module: Conformers Module: Scheduler Module: Species labels Mar 3, 2024

github-advanced-security bot found potential problems Mar 3, 2024

View reviewed changes

arc/main.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Mar 3, 2024

View reviewed changes

arc/scheduler.py Fixed Show fixed Hide fixed

arc/scheduler.py Fixed Show fixed Hide fixed

JintaoWu98 force-pushed the restart_multi_spc branch 2 times, most recently from d1aadc1 to 5ada2a6 Compare March 3, 2024 20:08

github-advanced-security bot found potential problems Mar 3, 2024

View reviewed changes

arc/scheduler.py Fixed Show resolved Hide resolved

arc/scheduler.py Fixed Show resolved Hide resolved

JintaoWu98 force-pushed the restart_multi_spc branch 5 times, most recently from d561230 to fe99b26 Compare March 4, 2024 12:11

JintaoWu98 removed the request for review from calvinp0 March 4, 2024 12:13

JintaoWu98 marked this pull request as draft March 4, 2024 12:14

JintaoWu98 requested a review from alongd April 15, 2024 17:13

Define output dict for multi species in main.py

916ff3d

main.py adds `output_multi_spc` as key in ARC.as_dict make output_multi_spc in restart_dict not None

JintaoWu98 marked this pull request as ready for review April 15, 2024 17:19

JintaoWu98 force-pushed the restart_multi_spc branch from fe99b26 to d2073a4 Compare April 15, 2024 17:19

JintaoWu98 added 2 commits April 16, 2024 22:27

Set ouput_multi_spc value true after running opt job

2bf0b61

Add output_multi_spc to test

5ca8ae1

JintaoWu98 force-pushed the restart_multi_spc branch from d2073a4 to 5ca8ae1 Compare April 16, 2024 19:28

calvinp0 approved these changes Apr 17, 2024

View reviewed changes

JintaoWu98 merged commit dfe93dd into main Apr 17, 2024
5 of 7 checks passed

JintaoWu98 deleted the restart_multi_spc branch April 17, 2024 15:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Define output dict for multi species in main.py #735

Define output dict for multi species in main.py #735

JintaoWu98 commented Feb 14, 2024 •

edited

Loading

codecov bot commented Feb 14, 2024 •

edited

Loading

alongd left a comment

alongd Feb 14, 2024

alongd Feb 14, 2024

JintaoWu98 Feb 15, 2024

calvinp0 Feb 15, 2024

calvinp0 Feb 15, 2024

JintaoWu98 Feb 16, 2024

calvinp0 Feb 16, 2024

JintaoWu98 Feb 16, 2024

calvinp0 Feb 16, 2024

JintaoWu98 Feb 16, 2024 •

edited

Loading

alongd commented Apr 16, 2024

calvinp0 commented Apr 16, 2024

JintaoWu98 commented Apr 16, 2024 •

edited

Loading

calvinp0 left a comment

Define output dict for multi species in main.py #735

Define output dict for multi species in main.py #735

Conversation

JintaoWu98 commented Feb 14, 2024 • edited Loading

codecov bot commented Feb 14, 2024 • edited Loading

Codecov Report

alongd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JintaoWu98 Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

alongd commented Apr 16, 2024

calvinp0 commented Apr 16, 2024

JintaoWu98 commented Apr 16, 2024 • edited Loading

calvinp0 left a comment

Choose a reason for hiding this comment

JintaoWu98 commented Feb 14, 2024 •

edited

Loading

codecov bot commented Feb 14, 2024 •

edited

Loading

JintaoWu98 Feb 16, 2024 •

edited

Loading

JintaoWu98 commented Apr 16, 2024 •

edited

Loading