chore: use poetry for python dependencies

[plexoos]

No description provided.

[rahmans1]

Just curious, is it not possible to handle this through spack alone? Also is this just for pyvista https://packages.spack.io/package.html?name=py-pyvistaqt ?

[plexoos]

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

[rahmans1]

Yes, Spack can be used to install python packages. Looks like they have it https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-pyvista/package.py but I think there always be a lag between what Spack has and the releases in pypi https://pypi.org/project/pyvista/

We can request a version update or I can add myself as a maintainer for that spack package. It usually requires a small update like spack/spack#41246 to have the version update if we require the most latest version.

[plexoos]

We can certainly consider Spack as a package manager for python... But I think for now a more pressing issue is that pyvista does not work on our GPU machine. Here I created an issue: #76
Could you please take a look?

[buddhasystem]

@plexoos so installing "poetry" and its paraphernalia is just another attempt to display pyvista on your Mac?

[plexoos]

No

[plexoos]

It's for non-Mac user like you

[rahmans1]

I think the major python packages are available through spack. I am tempted to branch of beta and see how well that works instead of adding poetry as a separate dependency management tool.

https://packages.spack.io/package.html?name=py-ipython
https://packages.spack.io/package.html?name=py-numpy

[plexoos]

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

[rahmans1]

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.

Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~# 

[rahmans1]

I don't have enough experience with Spack managing python packages, and I don't know how it deals with virtual environments... Poetry has no problem detecting an existing venv by conda or pip, or create a new one, and all of this pretty much automatically.
Anyway, here is how I suggest to test the current PR to see if the results can be visualized:

./esi-shell -v PR-72
# then inside the shell do
~# opticks-full-prepare
~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh 

Do I need to grab the .toml file separately or set another environment variable?

[rahmans@npps0 ~]$ ./esi-shell -v PR-72
==> Using esi-shell image: ghcr.io/bnlnpps/esi-shell:PR-72
==> Running docker container 8f7f78ada833
==> Setting up opticks environment. Please wait...
==> Found existing /home/rahmans/.opticks. Will use it

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

root@8f7f78ada833:~# opticks-full-prepare
=== opticks-full-prepare : START Fri May 24 18:03:40 UTC 2024
=== opticks-prepare-installation : generating RNG seeds into installcache
2024-05-24 18:03:40.161 INFO  [149] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.161 INFO  [149] [main@9] QCurandState::desc cs.num 1000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_1000000_0_0.bin
2024-05-24 18:03:40.164 INFO  [150] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.164 INFO  [150] [main@9] QCurandState::desc cs.num 3000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_3000000_0_0.bin
2024-05-24 18:03:40.167 INFO  [151] [QCurandState::init@37] scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin exists already : NOTHING TO DO 
2024-05-24 18:03:40.167 INFO  [151] [main@9] QCurandState::desc cs.num 10000000 cs.seed 0 cs.offset 0 cs.states 0 d_cs 0 scs.path /home/rahmans/.opticks/rngcache/RNG/QCurandState_10000000_0_0.bin
=== opticks-full-prepare : DONE Fri May 24 18:03:40 UTC 2024
root@8f7f78ada833:~# poetry run opticks/g4cx/tests/G4CXTest_raindrop.sh

Poetry could not find a pyproject.toml file in /home/rahmans or its parents
root@8f7f78ada833:~# 

Anyways. I figured it out. It just doesn't know where the .toml file is automatically. I have to launch it from the right folder or have it in path.

[rahmans1]

This system does not appear to be running an xserver.
PyVista will likely segfault when rendering.

Try starting a virtual frame buffer with xvfb, or using
  ``pyvista.start_xvfb()``
  warnings.warn(
opticks/g4cx/tests/G4CXTest_raindrop.sh: line 189:   302 Segmentation fault      (core dumped) ${IPYTHON:-ipython} --pdb -i $script
opticks/g4cx/tests/G4CXTest_raindrop.sh : ana error with script G4CXTest_raindrop.py

Is this the error you get now? I guess we will just need to modify the plotting script so that it doesn't try to launch the headless window with xfvb which is not in our system.

[buddhasystem]

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

[plexoos]

Sakib, please use the updated esi-shell from this pr

…

On Fri, May 24, 2024, 2:15 PM Maxim Potekhin ***@***.***> wrote: @rahmans1 <https://github.com/rahmans1> If the host npps0 is not running the X server, we need to make it do so. Can you do it? — Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGF6F7ZUE2VGQR77476UJDZD57TVAVCNFSM6AAAAABHY37CJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGEYTMNZWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rahmans1]

@rahmans1 If the host npps0 is not running the X server, we need to make it do so. Can you do it?

[rahmans@npps0 ~]$ startx
xauth: file /home/rahmans/.serverauth.1541623 does not exist

/usr/libexec/Xorg.wrap: Only console users are allowed to run the X server

xinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: server error
Couldn't get a file descriptor referring to the console

[rahmans1]

Sakib, please use the updated esi-shell from this pr
…

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

[plexoos]

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

curl -Os https://raw.githubusercontent.com/BNLNPPS/esi-shell/pr/poetry/esi-shell && chmod u+x esi-shell

[rahmans1]

I started with './esi-shell -v PR-72'. Is there something else that I am missing?

Please download the updated esi-shell script from the branch corresponding to this PR, like so:

I get the same error after getting the updated esi-shell script

[plexoos]

You reported two errors. The updated esi-shell addressed the first one where it could not find a pyproject.toml. Now, you don't need to install with poetry by hand.

The second one about "This system does not appear to be running an xserver ... " is because you are running remotely and X windows cannot be opened locally. If you have a direct access to a GPU machine (NVidia of course) with X11, this error should not appear. Unfortunately, I don't have such machine to test but that is what I expect.

Maxim @buddhasystem it seems like you have a local NVidia GPU. Could you try the test I suggested above to see if we get any visuals from it?