New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The future of NCL and the pivot to Python #64

Open
marylhaley opened this Issue Feb 5, 2019 · 13 comments

Comments

Projects
None yet
9 participants
@marylhaley
Copy link
Contributor

marylhaley commented Feb 5, 2019

This GitHub issue was created to give NCL users and the earth science community at large a chance to comment on the NCAR decision to adopt Python as the scripting language platform of choice for future development of analysis and visualization tools.

Please read this open letter to NCL users to understand what kind of impact this will have on the future of NCL.

We have added an FAQ to answer some of the common user questions we received after the announcement.

@marylhaley marylhaley changed the title Test GitHub Issue The future of NCL and the pivot to Python Feb 6, 2019

@marylhaley marylhaley pinned this issue Feb 6, 2019

@Peter-Gibson

This comment has been minimized.

Copy link

Peter-Gibson commented Feb 6, 2019

This saddens me - but is understandable. One thing I truly enjoy about NCL is the online documentation and its layout. What percentage of NCL functions will make their was into PyNGL - and over what timeframe?

@marylhaley

This comment has been minimized.

Copy link
Contributor Author

marylhaley commented Feb 7, 2019

@Peter-Gibson the plan is to determine which NCL computational routines are not already available in Python that are popular with users, and which NCL computational routines are not worth implementing either because they are already available in another Python module or they are no longer relevant. We are not ready to quote percentages right now as we are still new to this particular effort.

We are currently wrapping a suite of NCL computational routines into Python that are part of the Climate Diagnostics Variability Package, developed by Adam Phillips (one of the NCL Advisory Panel members). This package is a nice 'real-world' application of NCL which helps us identify some of the more popular routines first. We've identified about 40 functions in the CVDP that need to be wrapped into Python, while the remaining 80 to 90 ones we've identified as being already implemented in other Python packages, like numpy and xarray.

This 'CVDP Pilot Project', as we call it, is currently an internal GitHub project right now, but the scheduled timeframe for its first public release is later this summer. Here is a roadmap that provides more information. Note that these new routines will not be going into PyNGL, but rather in a separate not-yet-named Python module ("PyNCOMP").

After this point, we will look to our user base and our own example suite to help determine further functions to wrap.

We would love to hear from users who do plan to "pivot to Python" what NCL computational routines are the most critical for them. Communicating this information through this GitHub issue, or by submitting a new GitHub issue is an excellent way to vote on your favorite functions.

@dongli

This comment has been minimized.

Copy link

dongli commented Feb 7, 2019

NCL provides a one-stop environment for processing and visualizing data, especially the classified resources which we have gotten familiar with for years 👍. I hope these resources can be mapped to Python package equally (I have never gotten comfortable with Matplotlib 😂). Meanwhile, I hope we could only need to install one Python package instead a bunch of packages. Anyway, thanks for providing the horsepower for us!

@Try2Code

This comment has been minimized.

Copy link

Try2Code commented Feb 7, 2019

Dear NCL developers - thx for taking this important decision. It must have been a hard task to do with all the users and code heritage. I am looking forward to this because (1) I like the plotting features of NCL and (2) had the impression that NCL is somewhat limited in terms of language features (use of command-line arguments, user-defined data structures, key-value stores).

Python is certainly the way to go IMO from its reputation and wide-spread use in the field. But when it comes to python libraries, there area some lines I need to drop here: Please stick some of your concepts of how to describe a plot - don't let matplotlib do every thing, since it has some awkward features. In combination with Basemap, Cartopy or whatever related libraries this leads to strange patterns. NCL is the opposite: you get all from a single vendor (which I consider a good thing).
I just want to encourage you to stick to your concepts. I'd love to see a coherent approach for plotting climate and weather data esp. in combination with so powerful libraries like XArray and Dask.

Big thx for your support of the community!

@Peter-Gibson

This comment has been minimized.

Copy link

Peter-Gibson commented Feb 7, 2019

Hi Mary,

Thanks for the support and work on NCL over the years - its an amazing resource. I talked with other NCL users in our research group (who also use Python) and a few thoughts we collectively had:

  1. NCL's map plotting produces nicer plots with less effort and with the resources all in one place. This functionality into Python would be hugely valuable. I can make maps in Python look NCL-like but with many many more lines of code needed.
  2. NCL's functions for computing gradients (both using cfd and spherical harmonics) and NCL's functions for divergence, divergent wind components, vorticity, sf, vp are great because these are non-trivial to calculate - I don't think a full suite of these exist in Python - at least not all in one place.
  3. It would be great if NCL's website stayed online into the future (even if it wasn't updated). This is an highly valuable one-stop-shop resource for functions/examples/code.

Thanks again!

@wblumberg

This comment has been minimized.

Copy link

wblumberg commented Feb 7, 2019

Regarding thought number 2: windspharm (https://ajdawson.github.io/windspharm/latest/) might be what you're looking for. MetPy (https://unidata.github.io/MetPy/latest/index.html) has some pretty nice functions for calculating those too; they use finite differencing.

@billboos

This comment has been minimized.

Copy link

billboos commented Feb 8, 2019

One thing that has been nice about NCL is the ability to install it and go -- setting up the python environment can be a nightmare for new users, especially if you are trying to do so in an HPC setting. If, as part of this pivot, NCAR developers could provide some instructions, guides, or package management to help atmospheric scientists learning to use python in an HPC environment, that would be a major help. I switched from Matlab & NCL to python about a year ago, and I almost gave up in frustration after trying to get my Anaconda environment to work as a jupyter hub kernel on the clusters I use.

@Try2Code

This comment has been minimized.

Copy link

Try2Code commented Feb 8, 2019

maybe NCL could be shipped with a conda environment file to make this easier. It can be a nightmare to get libs to their correct versions ...

One thing that has been nice about NCL is the ability to install it and go -- setting up the python environment can be a nightmare for new users, especially if you are trying to do so in an HPC setting. If, as part of this pivot, NCAR developers could provide some instructions, guides, or package management to help atmospheric scientists learning to use python in an HPC environment, that would be a major help. I switched from Matlab & NCL to python about a year ago, and I almost gave up in frustration after trying to get my Anaconda environment to work as a jupyter hub kernel on the clusters I use.

@dongli

This comment has been minimized.

Copy link

dongli commented Feb 8, 2019

maybe NCL could be shipped with a conda environment file to make this easier. It can be a nightmare to get libs to their correct versions ...

And it would be nicer if we could install PyNGL through the standard pip.

@mvarma04

This comment has been minimized.

Copy link

mvarma04 commented Feb 8, 2019

I also see that scientific community is moving towards python. If anyone wants to take a transition from NCL to Python, I think it would be a great help for the NCL users to have some examples particularly relating to different types of plots. For the plotting purpose, I am totally depending on NCL right now. So, pages like this (http://www.ncl.ucar.edu/Applications/NCL_to_Python/) would be of great help.

@BarryHLynn

This comment has been minimized.

Copy link

BarryHLynn commented Feb 9, 2019

I just want to mention that I believe that the management and developers are quite underestimating the utility and potential of NCL. Almost every time I need to do something, I find a suitable example. Moreover, it is an incredibly logical language, often built on incredibly useful and simple to use commands (like conform). I am, at time, imply amazed at the care that went into building the logic of NCL and how the parts actually blend together to work seamlessly. NCL should not be abandoned -- it has a large, eager, and helpful user base with years of experience, willing to lend their ideas towards future development, while helping to maintain the integrity of the code. What's in Python that's not in NCL? There are plenty using Python -- let them use it and let them suffer from its inconsistencies. In the meantime, just build NCL where it's needed, taking the best ideas in Python -- but don't abandon a system that is and will be fundamentally better for its users (in the atmospheric sciences) -- for years to come.

@BarryHLynn

This comment has been minimized.

Copy link

BarryHLynn commented Feb 10, 2019

Hello:

I just wanted to make sure that the developers are aware of critical NCL functions (at least from my point of view).

For example,

  1. the calendar functions, inverse and regular.
  2. The ability to subset strings and obtain dates for example from the strings.
  3. The ability to use attributes to obtain times (e.g., from the GFS data).
  4. The ability to call shell commands from within NCL.
  5. The ability to parse variables in a list, e.g., CSV files read as single lines and then parsed into separate variables.
  6. The ability to "WRAPIT" Fortran programs.
  7. The ability to subset arrays and conform them.
  8. To be able to find min/max, and convert from 2d to 1d arrays for doing so.

Perhaps the biggest obstacle is learning how to do all of these things in Python, let alone making maps, overlays, panels.

It seems to me that it will take a while to create Python equivalent examples as exist online with explicit detail on the NCL pages.

@harrytheharris

This comment has been minimized.

Copy link

harrytheharris commented Feb 10, 2019

Thanks to all NCL devs, past and present, for a fantastic product: reliable, full-featured, and with a bottomless pool of assistance no matter your difficulty.
I'm sure this decision wasn't taken lightly; I'd been expecting something like this for some years and I'm glad to find my worst fears unrealised - that the existing implementation will survive, at least for now.
Having recently been obliged to work in R, as a result of inheriting someone else's work, I now view the challenge of new languages with trepidation; however, this gives me an excuse to play with Python without pressure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment