Condify pyproject.toml, add fileglancer as an editable pypi dependency via pixi, use pixi reinstall #158

mkitti · 2025-09-11T10:14:15Z

Make packages installable from conda. pypi packages are reduced from 95 to 1 (fileglancer itself)
Add fileglancer as an editable pypi install so that it is rebuilt on pixi reinstall
Replace task pip-install (pip install -e .) with pixi reinstall
Replace task pip-uninstall with pixi clean
Add PEP 735 dependency group for release

Current main branch


$ pixi clean
  removed /home/mkitti/src/fileglancer/.pixi/envs                                                                                                                                       

$ pixi install
✔ The default environment has been installed.

$ pixi list | grep pypi | wc -l
95

This pull request

$ git checkout mkitti-pixi-condify 
Switched to branch 'mkitti-pixi-condify'
Your branch is up to date with 'origin/mkitti-pixi-condify'.

$ pixi clean
  removed /home/mkitti/src/fileglancer/.pixi/envs     

$ pixi install
✔ The default environment has been installed.

$ pixi list | grep pypi | wc -l
1

$ pixi list | grep pypi
fileglancer                    0.6.1                                          pypi    (editable)

Another benefit is that a large part of dev-install, particularly pip install -e . can be replaced by pixi reinstall. If a source file is changed, pixi reinstall will rebuild the package. If nothing is changed, pixi reinstall will use its build cache.

Therefore, the pip commands have been replaced with pixi reinstall and pixi clean.

Previously, I thought of removing or replacing the task ext-clean, but I have left it in place. It may be less necessary now though due to pixi reinstall's behavior.

krokicki · 2025-09-14T16:39:26Z

@mkitti, I followed your instructions above and it worked as you showed, but afterwards when I tried either pixi run dev-watch or pixi run dev-launch-remote, I got this error:

Traceback (most recent call last):
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/bin/jupyter-labextension", line 6, in <module>
    from jupyterlab.labextensions import main
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab/__init__.py", line 8, in <module>
    from .handlers.announcements import (
    ...<3 lines>...
    )
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab/handlers/announcements.py", line 16, in <module>
    from jupyterlab_server.translation_utils import translator
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab_server/__init__.py", line 6, in <module>
    from .app import LabServerApp
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab_server/app.py", line 10, in <module>
    from jupyter_server.extension.application import ExtensionApp, ExtensionAppJinjaMixin
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_server/extension/application.py", line 17, in <module>
    from jupyter_server.serverapp import ServerApp
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_server/serverapp.py", line 34, in <module>
    import jupyter_client
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/__init__.py", line 3, in <module>
    from .asynchronous import AsyncKernelClient
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/asynchronous/__init__.py", line 1, in <module>
    from .client import AsyncKernelClient  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/asynchronous/client.py", line 8, in <module>
    import zmq.asyncio
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/__init__.py", line 52, in <module>
    from zmq import backend
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/__init__.py", line 30, in <module>
    raise original_error from None
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/__init__.py", line 25, in <module>
    _ns = select_backend(first)
  File "/groups/scicompsoft/home/rokickik/dev/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/select.py", line 40, in select_backend
    ns.update({key: getattr(mod, key) for key in public_api})
                    ~~~~~~~^^^^^^^^^^
AttributeError: module 'zmq.backend.cython' has no attribute 'PYZMQ_DRAFT_API'

mkitti · 2025-09-14T16:41:26Z

You might need a pixi clean if you are transitioning from an install where you used pip. pixi is unaware of libraries installed by pip.

allison-truhlar · 2025-09-15T17:30:19Z

@krokicki Running these two commands worked for me to then be able to use pixi run dev-watch:

pixi clean
pixi run dev-install

mkitti · 2025-09-15T19:59:36Z

To reproduce Konrad's issue:

./clean.sh
git checkout main
pixi run dev-install
git checkout mkitti-pixi-condify
pixi reinstall
pixi run dev-launch

AttributeError: module 'zmq.backend.cython' has no attribute 'PYZMQ_DRAFT_API'

``` (fileglancer) (base) kittisopikulm@kittisopikulm-wm1 fileglancer % pixi run dev-launch ✨ Pixi task (dev-launch in default): NODE_ENV=development jupyter lab --autoreload Traceback (most recent call last): File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/bin/jupyter-lab", line 6, in from jupyterlab.labapp import main File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab/__init__.py", line 8, in from .handlers.announcements import ( ...<3 lines>... ) File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab/handlers/announcements.py", line 16, in from jupyterlab_server.translation_utils import translator File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab_server/__init__.py", line 6, in from .app import LabServerApp File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyterlab_server/app.py", line 10, in from jupyter_server.extension.application import ExtensionApp, ExtensionAppJinjaMixin File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_server/extension/application.py", line 17, in from jupyter_server.serverapp import ServerApp File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_server/serverapp.py", line 34, in import jupyter_client File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/__init__.py", line 3, in from .asynchronous import AsyncKernelClient File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/asynchronous/__init__.py", line 1, in from .client import AsyncKernelClient # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/jupyter_client/asynchronous/client.py", line 8, in import zmq.asyncio File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/__init__.py", line 52, in from zmq import backend File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/__init__.py", line 30, in raise original_error from None File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/__init__.py", line 25, in _ns = select_backend(first) File "/Users/kittisopikulm/Documents/src/fileglancer/.pixi/envs/default/lib/python3.13/site-packages/zmq/backend/select.py", line 40, in select_backend ns.update({key: getattr(mod, key) for key in public_api}) ~~~~~~~^^^^^^^^^^ AttributeError: module 'zmq.backend.cython' has no attribute 'PYZMQ_DRAFT_API' ```

Running pixi clean or ./clean.sh seems to resolve the issue.

* Replace ext-clean task with `pixi reinstall` * Have pip-install and pip-uninstall commands depend on ext-clean to trigger `pixi reinstall`

…tall

mkitti · 2025-09-16T14:25:36Z

Konrad needed to run ./clean.sh before resolving his issue of AttributeError: module 'zmq.backend.cython' has no attribute 'PYZMQ_DRAFT_API'

krokicki · 2025-09-16T14:26:15Z

Thanks, it worked after I ran ./clean.sh.

I noticed that the new build is 50% slower for some reason. I tried each build twice, with a ./clean in between:

New:

real    3m7.535s
user    0m44.079s
sys     0m18.886s

Old:

real    2m13.242s
user    0m43.321s
sys     0m11.631s

New:

real	3m1.164s
user	0m44.733s
sys	0m18.280s

Old:

real	2m14.044s
user	0m43.231s
sys	0m11.170s

mkitti · 2025-09-16T15:22:22Z

What are you timing specifically?

I suspect that at the end of the day we may need less cleaning and that pixi reinstall may be sufficient in many cases.

krokicki · 2025-09-16T15:51:50Z

I'm timing the initial pixi run dev-install after a ./clean.sh.

This replaces `pixi reinstall` which reinstalls everything

…-pixi-condify

mkitti · 2025-09-27T00:18:14Z

I merged #182 into here for now because I could not push otherwise.

mkitti · 2025-09-27T00:28:25Z

I changed pixi reinstall to pixi reinstall fileglancer for the pip-install task.

Instead of reinstalling all of the packages it will only reinstall fileglancer which is more analogous to what pip-install previously did with pip install -e ..
Earlier in this PR, running pixi run dev-install after ./clean.sh would install all the packages twice including fileglancer.
On the main branch, pixi install followed by pip install -e . would install the dependencies twice but only install fileglancer once.

Also, I should note that dev-install is unnecessary after a ./clean.sh. That is you can run the following directly.

./clean.sh
pixi run dev-launch

This is because pixi install will install fileglancer itself as well as enable the extension.

time ./clean.sh && time pixi run jupyter labextension list
Cleaned up dev environment.

real	0m0.230s
user	0m0.001s
sys	0m0.007s
JupyterLab v4.4.7
/groups/scicompsoft/home/kittisopikulm/src/fileglancer/.pixi/envs/default/share/jupyter/labextensions
        fileglancer v0.8.0 enabled OK (python, fileglancer)
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)


real	1m37.220s
user	0m5.633s
sys	0m6.429s

Note that this is faster than @krokicki 's timings above.

The one thing you might need pixi run dev-install to do after a clean is pixi run node-install to run npm install. This is necessary for pixi run dev-watch to install run-p.

mkitti · 2025-09-27T02:16:35Z

On my Mac Studio the time is less than 20 seconds from clean to launch due to caching:

clean_to_launch.mov

mkitti · 2025-09-27T02:17:59Z

With an edit, it is just a few more seconds:

edit_reinstall.mov

krokicki · 2025-10-18T01:36:59Z

@mkitti Can you please resolve these conflicts so we can retest and merge this?

mkitti · 2025-10-21T15:32:31Z

@krokicki I have updated and resolved conflicts. There may be a need to go through the dependencies and see if we truly need them or not. The one I am least certain about is pydantic_settings_yaml.

mkitti · 2025-10-21T15:38:56Z

pixi run dev-launch seems to be encountering a lot of 404s... but I also get this on the main branch.

2025-10-21 11:37:39.792 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56215 [-] "GET /api/fileglancer/ticket HTTP/1.1" 404 - 1.67ms
2025-10-21 11:37:39.795 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56214 [-] "GET /api/fileglancer/notifications HTTP/1.1" 404 - 4.17ms
2025-10-21 11:37:39.795 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56212 [-] "GET /api/fileglancer/profile HTTP/1.1" 404 - 4.61ms
2025-10-21 11:37:39.796 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56217 [-] "GET /api/fileglancer/proxied-path HTTP/1.1" 404 - 4.70ms
2025-10-21 11:37:39.796 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56211 [-] "GET /api/fileglancer/file-share-paths HTTP/1.1" 404 - 4.75ms
2025-10-21 11:37:39.799 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56216 [-] "GET /api/fileglancer/ticket HTTP/1.1" 404 - 1.98ms
2025-10-21 11:37:39.801 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56215 [-] "GET /api/fileglancer/profile HTTP/1.1" 404 - 2.67ms
2025-10-21 11:37:39.801 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56212 [-] "GET /api/fileglancer/file-share-paths HTTP/1.1" 404 - 2.64ms
2025-10-21 11:37:39.802 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56217 [-] "GET /api/fileglancer/proxied-path HTTP/1.1" 404 - 2.87ms
2025-10-21 11:37:39.802 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56211 [-] "GET /api/fileglancer/notifications HTTP/1.1" 404 - 2.79ms
2025-10-21 11:37:45.211 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56251 [-] "GET /api/fileglancer/version HTTP/1.1" 404 - 1.32ms
2025-10-21 11:37:45.213 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56252 [-] "GET /api/fileglancer/central-version HTTP/1.1" 404 - 2.10ms
2025-10-21 11:37:45.215 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56251 [-] "GET /api/fileglancer/central-version HTTP/1.1" 404 - 0.77ms
2025-10-21 11:37:45.215 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56253 [-] "GET /api/fileglancer/version HTTP/1.1" 404 - 0.86ms
2025-10-21 11:37:48.966 | WARNING  | fileglancer.log:dispatch:83 - 127.0.0.1:56251 [-] "GET /api/fileglancer/notifications HTTP/1.1" 404 - 1.54ms

krokicki · 2025-10-21T17:54:16Z

@mkitti I tested the branch and everything works for me. You must have some old code in your local environment that is causing those old /api/fileglancer paths to be loaded.

I'm going to go ahead and merge the PR. Thanks a lot for your work on this!

mkitti added 5 commits September 16, 2025 10:09

Convert deps to use conda

f38a33f

Add fileglancer back as a editable package via pixi

09f0a3f

* Replace ext-clean task with `pixi reinstall` * Have pip-install and pip-uninstall commands depend on ext-clean to trigger `pixi reinstall`

Keep ext-clean, use pixi reinstall/clean instead of pip-install/unins…

3603a32

…tall

Remove pip as a pixi dependency

1c21f89

Update package-lock.json with fileglancer 0.7.2

716b02a

mkitti force-pushed the mkitti-pixi-condify branch from 0f8888a to 716b02a Compare September 16, 2025 14:12

mkitti added 4 commits September 26, 2025 18:32

Merge branch 'main' into mkitti-pixi-condify

0a9ad9b

Update lock files

e267cf1

Use pixi reinstall fileglancer for pip-install

66102bb

This replaces `pixi reinstall` which reinstalls everything

Merge branch 'mkitti-remove-unnecessary-files-dep-eslint' into mkitti…

4e82fd6

…-pixi-condify

mkitti and others added 3 commits October 10, 2025 11:44

Merge branch 'main' into mkitti-pixi-condify

c3a3930

Merge branch 'main' into mkitti-pixi-condify

71d5731

Merge branch 'main' into mkitti-pixi-condify

a025d74

mkitti added 2 commits October 20, 2025 16:57

Merge branch 'main' into mkitti-pixi-condify

da24837

Update pixi dependencies

ed58330

Merge branch 'main' into mkitti-pixi-condify

177c225

Merge branch 'main' into mkitti-pixi-condify

359f2bc

krokicki merged commit 138781c into JaneliaSciComp:main Oct 21, 2025
3 checks passed

Condify pyproject.toml, add fileglancer as an editable pypi dependency via pixi, use pixi reinstall #158

Condify pyproject.toml, add fileglancer as an editable pypi dependency via pixi, use pixi reinstall #158

Uh oh!

Conversation

mkitti commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current main branch

This pull request

Uh oh!

krokicki commented Sep 14, 2025

Uh oh!

mkitti commented Sep 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allison-truhlar commented Sep 15, 2025

Uh oh!

mkitti commented Sep 15, 2025

Uh oh!

mkitti commented Sep 16, 2025

Uh oh!

krokicki commented Sep 16, 2025

Uh oh!

mkitti commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krokicki commented Sep 16, 2025

Uh oh!

mkitti commented Sep 27, 2025

Uh oh!

mkitti commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkitti commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkitti commented Sep 27, 2025

Uh oh!

krokicki commented Oct 18, 2025

Uh oh!

mkitti commented Oct 21, 2025

Uh oh!

mkitti commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krokicki commented Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mkitti commented Sep 11, 2025 •

edited

Loading

mkitti commented Sep 14, 2025 •

edited

Loading

mkitti commented Sep 16, 2025 •

edited

Loading

mkitti commented Sep 27, 2025 •

edited

Loading

mkitti commented Sep 27, 2025 •

edited

Loading

mkitti commented Oct 21, 2025 •

edited

Loading