roadmap

[egayer]

Is there any roadmap available regarding Lets-plot ? is the plan to offer all the capabilities of ggplot2 through Lets-plot or is there parts of ggplot2 that will never be available in Lets-plot ?

[alshan]

No, we don't do roadmaps. We have a list of priority tasks in the beginning of each quarter and we do short-term planning every week depending on ongoing feedback and resources available.

As for ggplot2 API we are expecting to have most of it in some form in Lets-plot eventually.

For example currently trying to rework our coordinate systems in order to support polar coordinate system and various map projections.

Which in particular ggplot2 capabilities do you have in mind?

[egayer]

Thanks for your answer, my question was really to see if there was a way to know what-ish and when-ish to expect new features. I am glad to see that Lets-plot is growing and will bring more capabilities.

As for what capabilities I have in mind, actually you kinda answered !
I am personally interested in mixing rasters and shape files in UTM projections. I just came across the fact that right now, geo_map, geom_polygon only work in Lat/Lon. Even with a geodataframe with UTM coordinates, geo_map or geom_polygon transform everything into Lat/Lon. This makes impossible for the moment to plot any shape files (in geopandas geodataframes) on top of the geom_raster object using other coordinates than Lat/Lon. (when retrojecting the raster into WGS 84 Lat/Lon, then it works fine).
I am therefore very happy to read that you guys are working on the projection capabilities.

Now, I must say that I have never been a ggplot2 user. However Lets-plot is, to me personally, a great alternative of Matplotib. I am a big user of Matplotlib and since this is a discussion, I have a question regarding geom_image. This function accepts ndarray of Z, which is way more efficient than geom_raster which only accept (to my understanding) vectors for X, Y and Z. My question is then could geom_imageat some point behave like plt.imshow() (Matplotlib) when specifying a particular extent ([xmin,xmax,ymin,ymax]). With such extent parameter, plt.imshow automatically changes reference, not only the axis limits. Or is the way geom_image works making this impossible and the only way to plot a ndarray with specific limits and coordinates is to use geom_raster ?

[alshan]

Even with a geodataframe with UTM coordinates, geo_map or geom_polygon transform everything into Lat/Lon.

Yes, it does work this way currently. Whatever the original CRS is we retroject it to WGS84 lon/lat coordinates and then re-project coordinates according to selected coordinate system. coord_map() only supports the mercator projection at the moment. Others (coord_cartesian() etc.) just plot lon/lat coordinates without projecting.

At minimum, we can try to add for UTM projection to coord_map().

could geom_imageat some point behave like plt.imshow() (Matplotlib) when specifying a particular extent ([xmin,xmax,ymin,ymax]).

In ggplot2 this (or similar) is done by defining limits on scales x/y. By default limits are computed from the data but you can define different limits using xlim() or ylim() or "lims()" or via the parameter limits in scale_xxx().

I've just checked how it works with geom_image(): it works fine when limits are extended but fail if limits are narrowed (in relation to dimensions of the image).

Do you also require to be able to set the image coordinates? Currently its left-lower pixel is always at [0,0].

[egayer]

At minimum, we can try to add for UTM projection to coord_map().

That would be awesome ! thank you !

I've just checked how it works with geom_image(): it works fine when limits are extended but fail if limits are narrowed (in relation to dimensions of the image).

What do you mean by "extended limits" ? Is the following code failing because the limits "are narrowed" ?

Z=np.random.rand(10,15)
ggplot() \
   + geom_image(Z) \
   + scale_x_continuous(limits=[2,8]) \
   + scale_y_continuous(limits=[10,80]) \

It returns :

error on line 141 at column 18: Specification mandates value for attribute message
Error building plot:NullPointerException

Do you also require to be able to set the image coordinates? Currently its left-lower pixel is always at [0,0].

I have no special requirements regarding the image coordinates, I actually prefer Xmin and Ymin to be on the lover left corner of the image.

PS: I am currently learning Lets-plot (and then ggplot) while working on a research project. Some of my questions may appear candid but I hope it will help other Matplotlib users to understand and use Lets-plot (or ggplot)

[alshan]

What do you mean by "extended limits" ? Is the following code failing because the limits "are narrowed" ?

Yes, natural limits in your example would be x:[0,9], y:[0:14]. X-limits [2,8] is what I call "narrowed". Any wider limit like x:[-1, 10] would work.
"narrowed" limits should also work but with geom_image() specifically they don't at the moment.

I have no special requirements regarding the image coordinates, I actually prefer Xmin and Ymin to be on the lover left corner of the image.

In imshow() the parameter extent specifies "The bounding box in data coordinates that the image will fill."
This seems to make a perfect sense when we are speaking about something like satellite image. Do images you are using have a bounding box attached to them?
Actually I would very appreciate some kind of a simplified example of such geo-visualization in order to better understand the specifics and to use it as a reference.

[egayer]

Ok I see. An example would be more effective you are right. Let's say we have a DEM in Arc/Info ASCII Grid format (DEM.asc) for clarity (it would work with any geotiff or other raster format). Please find attached the notebook that shows how to open the file with gdal, then extract the numpy array and the georeferencing informations in order to create the plot with imshow() according to the geo-limits.

I understand that geom_image() plots a single band image (m x n array, a DEM typically) in gray only (which can be overcome by creating a m x n x 3 array), but the slowness of geom_raster() combined with the fact that it requires creating/mesgridding two other grids (of X's and Y’s) makes me think that changing the limits right in geom_image() would be a real bonus.

With such a feature and the ease with which geom_map manages geodataframes it allows me to consider making very good quality maps in a very efficient way.

Since I’m new in the ggplot world and I am learning Lets-plot, I'm testing the workflows I use with Matplotlib. The idea here is not to replace Matplolib but to take advantage of the best of both libraries while being able to create consistent figures/plots.
Example_EG.zip

[alshan]

making very good quality maps in a very efficient way.

Exactly, this was the intention all along. :)

Thanx for sharing the code, it's a great help.
Besides extent the support for cmap is also seems necessary.

[egayer]

Besides extent the support for cmap is also seems necessary.

Yes, that would be helpful to get colors on a simple m x n array.

I am currently working on modifying the plots on one of my webpage (see adress below) using Lets-plot. Nothing fancy for the 2 scatter plots on the page, but way more tricky for the Livemap using Folium which allows multiple layers. I didn't dive into live maps using Lets-plot yet, and the possibility to overlay raster or images. However, if you guys are interested in examples, I would be happy to let you know when I am finished modifying this webpage using Lets-plot. I would also be happy to share the Lets-plot figures of what am working on right now (using the full capabilities of Lets-plot :)) but it is not published yet so it needs to wait.

Here for info the adress of what I am currently modifying using Lets-plot (right now it is still from Folium and Plotly):
http://www.ipgp.fr/~egayer/Research_Reunion/

[alshan]

It's very cool you've chosen to use Lets-Plot instead of Plotly/Folium!

I didn't dive into live maps using Lets-plot yet, and the possibility to overlay raster or images.

Livemap in Lets-Plot does also support multiple layers of overlays but for the moment they are only: point, path, polygon, tile (and derived like contour etc.). If you require the image overlay then we need to work on this capability as well.

if you guys are interested in examples

Absolutely! Please keep us updated.

[alshan]

Hi Eric!
I've got some progress with respect to the extent parameter in geom_image(), check this out: https://nbviewer.org/github/JetBrains/lets-plot/blob/master/docs/f-22d/image_extent.ipynb

Btw, have you ever considered using PyCharm for some of your projects? There is Lets-Plot in SciView plugin for PyCharm allowing visualization right in the IDE. However, the "SciView" option is only available in "pro" edition.

[alshan]

Hi @egayer! In the recent release 2.5.1 we've added features we discussed here.

Note that geom_image() was renamed to geom_imshow() and now works the same way the matplotlib imshow() does in terms of extent, cmap, vmin and vmax parameters.

We've added support for UTM and other CRS-s as well, via the use_crs parameter.

In short, when you draw a DEM image and a GeoDataFrame on the same chart, you will pass EPSG code of the image's CRS to geom_map() (or other relevant geometry layer) via the use_crs parameter. This way both layers will be rendered using the same coordinate system.
You will find an example in this example notebook.

There are also few other examples in the CHANGELOG.

Why don't you give it a try? )

Note, image overlay on live-map is not yet supported, may be next time.

[egayer]

Awesome ! thank you so much for your very quick dev !
Of course, I will try all these ASAP and will give you feedbacks, but from what you described, it already looks good !
I'll keep you posted

[egayer]

@alshan, after a quick try mixing a DEM, a shape file and some x,y coordinates all the same figure, everything works perfectly !
geom_show() is super fast, the extent option does what it does in plt.imshow(), and plotting shape files and other geometries such as geom_point() with the corresponding x,y, on top of a DEM is super easy now. I only used UTM but the use_crs works fine.

I did try it quickly so I can give a quick feedback, however I came across the fact that geom_imshow() doesn't like NaN because it "cannot convert float NaN to integer". I don't know how geom_image() used to deal with NaN and I am pretty sure I am missing something on how to specify no_data value in lets-plot in general, but I just mention this here as a quick feedback.

I will test some other stuff and I'll keep you posted, thanks again !

[alshan]

Are NaNs typical for DEM/spatial images?

In general in lets-plot, positional "no data" values are dropped, and not positional (color, fill) are replaced with na_value specified via scale_xyz() function.
But this doesn't apply to geom_imshow() because it doesn't use ggplot scales for rendering.

[egayer]

sorry, it was too short !
here is the link again

https://dl.ipgp.fr/5sgf18

[alshan]

Thanks! As for geom_label(), try hjust="left", nudge_x=... instead of position=position_nudge(x=0.5e4)

[egayer]

do you want me to change it in the notebook ?

[alshan]

No, I'm good with the variant you've uploaded. Simply noticed that you had small issues with label positioning.

[egayer]

OK ! I'll use what you suggested from now on

[alshan]

Hi @egayer , just my update on the progress.
The issue with NaN-s is now solved. Here is the demo.

Also, I was able to many-folds improve geom_imshow() performance so that is can handle large DEM files pretty well.

[egayer]

Hi @alshan, thanks for the update, the demo looks great ! Thanks you again for your quick dev, I can't wait the next release to play with it !

PS: also, that kind of demo is really useful especially for small tricks such as the geom_rect() for water. My first though would have been to work on the color scale for the value in ocean (and not using NaN-s), which would have been way too complicated !

[alshan]

I hope, one day many other people will learn many more mapping tricks from your publication at Medum or alike. ;)

[alshan]

Hi, you already can try lets-plot 3.0.0rc1: geom_imshow() is ready for industrial use (high performance, new parameters). But if something is still missing there is yet some (little) time to fix till the release of 3.0.0.

Some "breaking" changes: Python 3.6 is no longer supported, geom_livemap() interface was cleaned-up but AFAIK this shouldn't affect you.

[egayer]

Awesome ! I'll try the new version in the coming days and will let you know if I see something missing, but I doubt it, geom_imshow() of 2.5.1 is already working very well.
Thanks for the update !

[alshan]

Hi @egayer , could you share some sources of DEM files and shape-files?
We are planning to add the "Static Maps" section to our documentation but UTM DEMs aren't easy to find, as well as matching Shape-files .
Thanx.

[egayer]

Me too :) I've just submitted the issue : #780.

haha ok cool but honestly no rush :)

Gosh, we don't even have a single example with Polaris :( Thanx for the tip!

First I felt like there would be no need for such an example, but I couldn't resist to make one little notebook that we could name "Dict, Pandas and Polars". I just took your examples for geom_point() to show how to make a plot (i) with dictionaries, (ii) with Pandas Dataframes generated from dictionaries and from .csv files, (iii) same with Polars Dataframes.

DictPandasPolars.ipynb.zip

[alshan]

Great, thank you! We will definitely add it to our documentation.

[egayer]

Good, glad it helps.

I have a question regarding geom_imshow() but I feel like it may have to do with theme() instead. As we discussed a while ago you made the extent argument so we can use it to use the actual limits of a DEM (or any image as you explained in the doc). I always thought it was working well with DEM but today I tried to fit the axis to a DEM (or image) and I can't wrap my head around it.
Using theme_bw() or theme_classic() I get a bounding box around the actual data. I double checked and it looks like it is not at all a problem with the extent argument. I also tried to use panel_background=element_rect(size=1) in theme() but I still get a gap between the edge of the image and the axis of the plot. It doesn't seems to be the case for other x,y graph that is why I first though it was because of extent.
Do you have any idea how we could align the axis (or panel_background=element_rect()) to the edge of the DEM/image ?

[alshan]

The gap you see between the data and axis is likely the effect of positional scales "expand".
Try to set the multiplicative and additive "expand" to zero:

... + scale_x_continuous(expand=[0,0]) + scale_y_continuous(expand=[0,0])

[egayer]

Thanks ! I forgot about these parameters, it works perfectly.

[alshan]

Hi @egayer , It took a while but finally your "data guide" made it to production :) See "example notebook" in the "Data" section: https://lets-plot.org/pages/charts.html

Thanx for contribution!

[egayer]

Hi @alshan, awesome ! happy to help. I didn't find enough time yet to suggest something about the mapping stuff but I am still interested, so I may propose something in the (near) future

[egayer]

Hi @alshan,
I have a little and stupid question I am sure, but I couldn't find the answer anywhere. I am wondering how to ask Lets-Plot to "break" lines or path when it encounters a NaN (it is related to what I mentioned earlier when I asked you guys if geom_imshow() could plot NaNs as transparent pixels ;)).

I tried to geom_path() values containing "None" instead of np.nan, but Pandas transforms "None" into NaNs... (I didn't try with Polars though). I also double checked with ggplot2 who seems to handle it with an argument in ggplot() like na.rm=True but couldn't find it in the Lets-Plot ggplot() function.

for example I would like to plot these 3 lines as independent lines (variable1) and coloured by variable2

df1=pd.DataFrame({'x':[1,2,3,1.5,2,4,0,3,4.5,6], 'value':[21,12,3.3,35,20,14,30,23,14.5,6], 'variable1':[1,1,1,2,2,2,3,3,3,3], 'variable2':[10,10,10,10,10,10,20,20,20,20]})

ggplot(df1) + \
   geom_point(aes(x='x',y='value',color='variable2'), size=3)+ \
   geom_path(aes(x='x',y='value', color='variable2'),size=1)

but there is no break between the different lines. I thought one trick would be to insert NaNs to separate the lines :

df2=pd.DataFrame({'x':[1,2,3,1.5,2,4,np.nan,0,3,4.5,6], 'value':[21,12,3.3,35,20,14,np.nan,30,23,14.5,6], 'variable1':[1,1,1,2,2,2,np.nan,3,3,3,3], 'variable2':[10,10,10,10,10,10,np.nan,20,20,20,20]})

ggplot(df2)+ \
   + geom_point(aes(x='x',y='value',color='variable2'), size=3)+ \
   + geom_path(aes(x='x',y='value', color='variable2'),size=1)

but the different lines are still joined :(

thanks for your help and directions !
Eric

[alshan]

Hi Eric, you need to either use the "group" aesthetic: ..., color='variable2', group='variable2'

or make 'variable2' a discrete variable:

• option 1: use strings as values
• option 2: use numerics as values but annotate with 'as_discrete()':

from lets_plot.mapping import as_discrete
...
...color=as_discrete('variable2')

As for NaN-s , we just drop all the data where x or y is NaN.

[egayer]

Awesome ! As always, great solution, it works perfectly, thank you very much ! (I just learned how to use “group” ! But didn’t try the other suggestions yet)

Just a little comment regarding the NaNs, my question caame from my experience with matplotlib.

for example :
plt.plot([1,2,3,np.nan,6,7], [12,27,39,np.nan,65,71])

Would result in 2 segments (same color of course)

To get this result with geom_path() from these x and y, shall we modify the dataset so we can use “group=“ ? Or maybe your option 1 and/or 2 would work ?

[alshan]

What you are saying regarding the alternative handling of NaN-s makes a perfect sense - please feel free to file a feature request.

As for the grouping, group=var is an explicit way to tell lets-plot which variable should be used as a grouping variable. The other two options I proposed are essentially the same but the grouping occurs implicitly in this situation (by factor levels of discrete variable). In all cases you will need a variable to group by.

However, there is yet another option: you can create a GeoDataframe containing segments of your path as separate Lines / Multi-Lines. (https://lets-plot.org/pages/geopandas.html)
This might fit your case better maybe.

[egayer]

ok cool so I just post the request in the issue section ?

[alshan]

exactly right )