NPE1Adapter Part 3 - caching of adapter manifests

[tlambert03]

Breaking up #86 into three steps. This is part 3 (but it contains #124 and #125 which can be reviewed first)

This PR adds local caching of adapter manifests after contributions have been imported and indexed. This is the thing that will really make napari much faster on load time.

• This lazily loaded manifest is used for the duration of the session, and also cached for use in the next session
• caching will not be done for packages that are installed in editable mode.
• the npe2 cache command provides additional ways to query/modify the cache
• cache can be cleared by using npe2 cache --clear.
• Cache can be ignored per session by starting napari with NPE2_NOCACHE=1

[codecov]

Codecov Report

Merging #126 (0ae93cc) into main (39f00b9) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #126   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           25        25           
  Lines         1650      1719   +69     
=========================================
+ Hits          1650      1719   +69     

Impacted Files	Coverage Δ
npe2/_command_registry.py	100.00% <100.00%> (ø)
npe2/cli.py	100.00% <100.00%> (ø)
npe2/manifest/_npe1_adapter.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39f00b9...0ae93cc. Read the comment docs.

[nclack]

Testing this out I ran into something I don't understand. This is on a py39 env with napari main, this pr and napari_animation (which is npe1) installed. If I run python and import napari_animation I get:

% python
>>> import napari_animation
/Users/nclack/src/CZI/npe2/npe2/manifest/_npe1_shim.py:80: UserWarning: Failed to detect contributions for npe1 plugin 'napari-tools-menu': cannot import name 'get_settings' from partially initialized module 'napari.settings' (most likely due to a circular import) (/Users/nclack/src/CZI/napari/napari/settings/__init__.py)
  warnings.warn(

which seems strange? I'm out of time tonight, but I'll try to reproduce in the morning with a clean env.

On the success side, I can see e.g. napari-ndtiffs get cached. I really like that there's cli support.

[nclack]

I ran into an issue and got confused. Will pick this up in the am.

[tlambert03]

I'll take a look soon. I'll remind you that napari tools menu is not really a plugin, but rather a monkey patch that doesn't have hook implementations, so all bets are off for that one .

[nclack]

reproduce in the morning with a clean env.

Did not reproduce. Works fine on a clean env.

[nclack]

Looks good. There was one spot where you hardcode the length of SHIM_NAME_PREFIX and I left a couple of questions.

One question I had was how to document some of the inferences that get made when converting plugins (like categorizing a theme as dark/light mode based on luma). Or extracting glob patterns for readers. I feel like that's a future discussion though.

[nclack]

I'm not quite sure what the TODO is referring to. I'm not sure I would factor anything out of this function as is. Should probably avoid the 13 - replace with len(SHIM_NAME_PREFIX) etc.

[tlambert03]

/Users/nclack/src/CZI/npe2/npe2/manifest/_npe1_shim.py:80: UserWarning: Failed to detect contributions for npe1 plugin 'napari-tools-menu': cannot import name 'get_settings' from partially initialized module 'napari.settings' (most likely due to a circular import)

wasn't able to repeat in a clean environment with napari, npe2, and napari-animation... but if I pip install napari-tools-menu i do get it. will dig

[tlambert03]

wasn't able to repeat in a clean environment with napari, npe2, and napari-animation... but if I pip install napari-tools-menu i do get it. will dig

yeah, this circular import is because napari-tools-menu imports the private napari._qt module here. sorry, that's a "will not fix" 😂

[tlambert03]

merging!

This by no means implies that we can't iterate / refactor / cleanup moving forward. but it will make it a bit easier to get this in and play with it a bit