Skip to content

Commit

Permalink
Add util for checking stack version in notebooks
Browse files Browse the repository at this point in the history
  • Loading branch information
@mfisherlevine
mfisherlevine committed Mar 4, 2022
1 parent c701331 commit 84a0ab1
Showing 1 changed file with 17 additions and 0 deletions.
17 changes: 17 additions & 0 deletions python/lsst/rapid/analysis/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.

import eups
import numpy as np
from scipy.ndimage.filters import gaussian_filter
import lsst.afw.detection as afwDetect
Expand Down Expand Up @@ -292,3 +293,19 @@ def makeQuery(dataId):
spectractionQuantum = quanta[3]
result = butler.get(spectractionQuantum.outputs['spectraction'][0])
return result


def checkStackSetup():
ep = eups.Eups()

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

We try very hard not to depend on eups in any of our code.

The public API we have for this is lsst.utils.packages.

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022 •
edited

Author Contributor

Ah, nice, I've not played with that much. I see I can do the 2nd half easily with
for package, pathAndVersion in packages.getEnvironmentPackages().items():
path = pathAndVersion.split(':')[1].split("@")[0]
print(f"{package} setup at {path}")
but can't easily see how to get the weekly tag. Any idea how to do that bit?

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022

Author Contributor

(and in before comments about regex)

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022 •
edited

Firstly getEnvironmentPackages only reports locally setup packages. This is because the system tries not to tell you by default about packages you haven't imported (if you've imported afw then it will report the afw version via the standard python versioning system). It would be easy to add a flag to getEnvironmentPackages that would report all setup packages (at the moment they are deliberately skipped but tweaking that is a couple of lines).

Hardcoding lsst_distrib seems wrong but maybe you want something like:

diff --git a/python/lsst/utils/packages.py b/python/lsst/utils/packages.py
index 6f57f57..e5d06d3 100644
--- a/python/lsst/utils/packages.py
+++ b/python/lsst/utils/packages.py
@@ -193,6 +193,7 @@ def getEnvironmentPackages() -> Dict[str, str]:
     # is clean).
     for prod in products:
         if not prod.version.startswith(Product.LocalVersionPrefix):
+            packages[prod.name] = prod.version + " (" + " ".join(prod.tags) + ")"
             continue
         ver = prod.version

(which would only trigger if a new flag was added to report all environment packages and so wouldn't contaminate the normal output). You would get something like:

 'meas_modelfit': 'g8cdf404ddd+fc255244d7 (b5962)',
 'obs_base': 'geff6ec3125+e9c81dd162 (b5962)',
 'obs_cfht': 'g2b6e816441+be9ed484c0 (b5962)',
 'obs_decam': 'g8f5724d3ed+caa29b6fbd (b5962)',

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022 •
edited

Author Contributor

getEnvironmentPackages only reports locally setup packages

Perfect, that's exactly what I'm trying to do here.

But there's two separate problems I'm trying to solve:

  1. report all the locally setup packages. That's working perfectly already.
  2. Just get the weekly tag that's currently setup, regardless of whether anything's changed.

Do you think this can be done?

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

If you can explain to me how you find the weekly tag that's currently setup then we can make that work. The code can't rely on lsst_distrib itself being set up. I thought the code I put up above would do what you want though. Won't that report the weekly tags? (my example above was not a weekly because it's a random build I had checked out)

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022 •
edited

Author Contributor

The code can't rely on lsst_distrib itself being set up

Why is that?

Currently, I just do:

for tag in ep.findSetupProduct("lsst_distrib").tags:
        if tag != "current":
            print(f"You are running {tag} of lsst_distrib")

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

lsst.utils.packages has no idea about lsst_distrib. It can return the tags for all the packages (as I show above) and you can decide that you are only interested in the lsst_distrib entry.

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

I'm trying to understand if a variant of that diff would work for you or if I'm missing the point completely.

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022

Author Contributor

OK, I think it's just a bug, gimme a sec to poke it, curently that's making it print all the packages all the time whether local or not...

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

Are you testing my patch or something? My patch isn't complete (it returns all packages -- the patch doesn't include the new "include_all" parameter). I was using it as a demonstration.

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022

Author Contributor

Was doing that with tweaks, yeah. Sure, something like that should work. Still not quite sure what the problem with the lsst_distrib part is though. I guess we update that utils.packages stuff, and then I do the finding the LOCAL: part myself to filter, and then just don't do that for the lsst_distrib component. So the plan would be to have it take an option to always return everything, along with the weekly tag? If so then yes, that sounds perfect.

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

My point is that utils can't special case lsst_distrib but you can absolutely report the lsst_distrib version by getting it out of the dict. Okay. I'll do a quick ticket then.

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022

Author Contributor

Ah, wonderful, yes, now I see what you mean, clearly you can't special case lsst_distrib there 👍

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 4, 2022

Author Contributor

Any many thanks, that's great 🙏

This comment has been minimized.

Sign in to view

Copy link
@timj

timj Mar 4, 2022

With lsst/utils#117 at NCSA:

$ python -c 'import pprint; import lsst.utils.packages as P; pprint.pprint(P.getEnvironmentPackages(include_all=True))' | grep lsst_distrib
 'lsst_distrib': 'g64fc59b30a+43385bd5fa (w_2022_10)',

This comment has been minimized.

Sign in to view

Copy link
@mfisherlevine

mfisherlevine Mar 29, 2022

Author Contributor

Works great, I've pushed a change to make use of this. Thanks again!
for tag in ep.findSetupProduct("lsst_distrib").tags:
if tag != "current":
print(f"You are running {tag} of lsst_distrib")

products = ep.findProducts(version='LOCAL:*', tags='setup')
if products:
print("\nLocally setup packages:")
print("-----------------------")
for product in products:
print(f"{product.name} setup at {product.dir}")
else:
print("\nNo packages are setup locally.")

0 comments on commit 84a0ab1

Please sign in to comment.