Update package versions for v. 1.2.3

[xylar]

Updated versions:

• cdp removed
• e3sm_diags ==1.4.0
• nco ==4.7.7
• processflow ==2.1.0

Added:

• pyflann
• scikit-image

[xylar]

@sterlingbaldwin, I'm working on a new version of e3sm-unified. It seems like processflow depends on acme_diags, which depends on cdp=1.1.0. I think processflow should, indstead, depend on e3sm_diags, which should depend on the same version of cdp that I have above. Could you make that fix and perhaps update the minor version of processflow accordingly?

@zshaheen, let me know if what I say above isn't correct.

[xylar]

@zshaheen, is this the desired version of e3sm_diags?

@jhkennedy, is there an update for livvkit?

@czender, nco==4.7.7, correct?

@doutriaux1, any updates to CDAT other than cdp==1.4.0 (which is required for compatibility with e3sm_diags)?

[sterlingbaldwin]

@xylar I talked to Zeshawn about this a bit ago actually. The latest code on processflow master has e3sm_diags instead of acme_diags, but I havent built a new version yet for conda. I'll take care of that and publish a new minor version update.

[doutriaux1]

@xylar I think cdms2 was upped to 3.0.1 @dnadeau4 is that correct?

[zshaheen]

@xylar Yes, version 1.4.0 is the correct version of e3sm_diags.

For E3SM, the only reason cdp is in there is due to e3sm_diags. When I release a new version of e3sm_diags, I report the changes on this Confluence page. So when a cdp is upgraded for e3sm_diags, should I still update it there? I think just having whatever version of e3sm_diags we need would be fine, since it'll automatically get the appropriate version of cdp.

[czender]

Yes, NCO 4.7.7 is correct

[xylar]

@doutriaux1, cdm2==3.0.1 was already set in 1.2.2: https://github.com/E3SM-Project/e3sm-unified/pull/26/files#diff-08ec1c8c69bdd6ee90a1634993202eedR12

[xylar]

@zshaheen wrote:

I think just having whatever version of e3sm_diags we need would be fine, since it'll automatically get the appropriate version of cdp.

Okay, so I should just take cdp out as a dependency and let e3sm_diags decide, right?

[zshaheen]

@xylar, Yes, we should be fine with removing cdp as a dependency in the e3sm unified env.

[jhkennedy]

@xylar 2.1.6 is still the right version for LIVVkit -- 2.2.0 is still a little ways off.

[xylar]

@sterlingbaldwin, I just tried the latest upload (processflow 2.1.0) but the hdf5 version is not compatible with conda-forge (e.g. NCO 2.7.7). Could you rebuild with hdf5 1.10.3? Perhaps 2.1.0 wasn't ready to test in any case, but I figured I'd give it a try.

[sterlingbaldwin]

I dont tag specific version of anything besides Globus, which I dont think should affect the hdf5 version. Even though I dont use it directly, I'll try rebuilding with hdf5 == 1.10.3 and see if there are any conflicts.

[xylar]

@sterlingbaldwin, hmm, I'm still seeing conflicts but this time nothing to do with hdf5. I just see:

conda.exceptions.UnsatisfiableError: The following specifications were found to be in conflict:
  - nco==4.7.7
  - processflow==2.1.0
Use "conda info <package>" to see the dependencies for each package.

Maybe this is a different build of processflow 2.1.0? You said you don't specify a version of nco?

[sterlingbaldwin]

My requirements read as follows:

    - pip
    - python
    - setuptools
    - nco
    - ncl
    - e3sm_diags
    - e3sm_to_cmip
    - sqlite
    - peewee
    - configobj
    - beautifulsoup4
    - lxml
    - paramiko
    - jinja2
    - globus-sdk ==1.1.1
    - globus-cli ==1.1.2
    - click
    - jmespath
    - requests

I havent done a rebuild yet, so if anything changed it wasnt because of me. Conda can be weird sometimes, for example I had to add a couple things like jmespath that I dont need, and are listed as dependencies of globus, but once or twice I had an install just not bring them in.

[xylar]

Yeah, frustrating! I'm going to try to build processflow myself and upload to my own channel to see if I can get it working. I'll keep you posted.

[xylar]

@sterlingbaldwin, Okay, after a bunch of mucking around, I think the problem is that nco=4.7.7 was built with hdf5=1.10.3 but ncl=6.5.0 was build with hdf5=1.10.2. I'm not sure we can proceed until ncl has been updated. I got the impression from @ocefpaf that we shouldn't expect packages to get rebuilt with the latest pinnings too immediately because conda-forge developers are focused on compilers at the moment.

[sterlingbaldwin]

That would make a lot of sense. NCL has always been a little slow to rebuild. Ideally I would like to drop support for AMWG and just support e3sm_diags which doesnt have any of these problems, but for the short term Im not sure what the solution would be. What I did in the past was just require that users install ncl for themselves which is a bit of a pain without using the conda version. Do you have any suggestions? I could just go back to requiring users that want to use AMWG to handle their own dependencies for it.

[xylar]

Hmm, I think it might be worth waiting to see if NCL gets updated on conda-forge. I'm not aware of a pressing need to have the latest version of any of the packages that would be updated here (@czender or @zshaheen, please correct me if I'm wrong). It would be better to wait for a clean solution than to rush a difficult one.

[milenaveneziani]

I don't personally use NCL, but I know there are other people that do, outside of the AMWG package.

[xylar]

Yes, I'm also aware of some NCL users so best to make sure it's available, I think...

[milenaveneziani]

It probably doesn't need to be updated often, unless there are specific dependencies.

[zshaheen]

@xylar @sterlingbaldwin, for us, there's no urgent need to have the latest e3sm_diags available via the unified env.

[xylar]

@milenaveneziani, NCL depends on an hdf5 and that's the conflict we're having with NCO (hdf5 1.10.2 vs. 1.10.3). If not for that conflict, I would agree there's no urgent need to update.

[czender]

I am unaware of any urgent reason to upgrade the HDF, GSL, or ESMF libraries upon which NCO depends. They could also be downgraded without noticeable harm. The key library to ensure is netCDF >= 4.6.1. Nothing else is as crucial.

[xylar]

@czender, I guess you're right that we could do a build with NCO 4.7.7 but hdf5 1.10.2 and that might solve the problem for now. Let me see what I hear back from the NCL folks and we'll do that if need be.

[xylar]

Closes #25