Two projects, job submission

[JaGeo]

I have two projects. When I submit a job with the wrong project (the demon of this project is not running), the job gets submitted to the queue.

Even when I take the correct project name for the submission, the workflow shows up in both projects when typing:

jf -p project2 job list
jf -p project1 job list

I am not sure this is intended. (I am using the same database but have specified different collections for outputs)

Related or maybe not related: then the data retrieval from the remote runner fails.

[gpetretto]

In principle this is expected. In our idea two different projects may be more or less like two different setups of fireworks with two different launchpads. So, inside each document in the queue Store there is no information about which project it belongs to, as it is basically assuming that all the jobs will belong to just one project.
I am not surprised you get weird behaviour, since you will have two runners working on the same DB collection. There are several things that could go wrong in that case.

There are two options:

1. use two different databases for the two different projects
2. if you want to keep a single DB also for the queue store you can specify the collection names

    queue:
      store:
        '@module': maggma.stores.mongolike
        '@class': MongoStore
        database: DB_NAME
        collection_name: jobs_project1
        ...
      flows_collection: flows_project1
      auxiliary_collection: jf_auxiliary_project1

   and the same for project2

Would this be fine with you?

On the other hand, since the outputs are mainly based on jobflow's uuid, in principle you could use the same collection for the outputs.

In any case, thanks for raising the issue. I will clarify this in the documentation.

[JaGeo]

Thanks, @gpetretto , for the clarification! I will try your suggestion. I think I will try option 2.

Just do add: only one daemon was activated. Thus, I did not expect these side effects.

[gpetretto]

With only one daemon activated it is indeed weirder that you got some error. Do you maybe still have the exact error message?

[JaGeo]

It raises a remote error and cannot fetch the queueerror from the db.

I am also using two remote workers and I am also not sure this is connected. Would need to do a lot of testing that I don't have time for at the moment

[JaGeo]

In any case, I will get back to this

[JaGeo]

I tried again with the suggested settings. This does not work as well.
It seems that my connection is okay to submit but not to retrieve data. I use the store-interactive branch still.

[JaGeo]

This also does not work:

(2025-03-05-Lobster-school) jgeorge@sv2218:/hpc-user/jgeorge/PycharmProjects/2025-03-05-Lobster-school$ jf -p lobster job queue-out  1
The selected project is lobster from config file /home/jgeorge/.jfremote/lobster.yaml
[15:19:36] WARNING  Error while trying the authentication with all the public keys available: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported    
                    algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>]). This may be due
                    to the format of one of the keys. Authentication will proceed with interactive prompts                                                                                                         
An Error occurred during the command execution: AuthFailure 
Interactive(user='di82tut') -> Authentication timeout.

It never asks me to login.

[JaGeo]

Potentially, I have done something stupid with the setup. Will reinstall later and report back.

Sorry!

[gpetretto]

This also does not work:

(2025-03-05-Lobster-school) jgeorge@sv2218:/hpc-user/jgeorge/PycharmProjects/2025-03-05-Lobster-school$ jf -p lobster job queue-out  1
The selected project is lobster from config file /home/jgeorge/.jfremote/lobster.yaml
[15:19:36] WARNING  Error while trying the authentication with all the public keys available: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported    
                    algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>]). This may be due
                    to the format of one of the keys. Authentication will proceed with interactive prompts                                                                                                         
An Error occurred during the command execution: AuthFailure 
Interactive(user='di82tut') -> Authentication timeout.

It never asks me to login.

I should have fixed this in the interactive and interactive-store branches.

I will wait for your feedback before investigating the rest of the issue further.

[JaGeo]

It now works again. I think I will close this for now, as I did not investigate systematically. I am now using the default collections again and deleted the other project.

However, pointing out again that different databases/or different collection names for all required collections are needed if one wants to use two different databases.

(Sorry for the mess. I wanted to do something quickly and ended up doing everything 5 times... )