Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSM tiling #440

Merged
merged 48 commits into from
Sep 22, 2023
Merged

OSM tiling #440

merged 48 commits into from
Sep 22, 2023

Conversation

dalmijn
Copy link
Contributor

@dalmijn dalmijn commented Jul 12, 2023

Issue addressed

Fixes #345

Explanation

Created tiling function by starting from the bottom in terms of zoomlevel. This starting level is user defined or defaulted to 5.

Checklist

  • Updated tests or added new tests
  • Branch is up to date with main
  • Tests & pre-commit hooks pass
  • Updated documentation if needed
  • Updated changelog.rst if needed

Additional Notes (optional)

Add any additional notes or information that may be helpful.

@dalmijn dalmijn changed the title Osm tiling OSM tiling Jul 12, 2023
Copy link
Contributor

@DirkEilander DirkEilander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together @B-Dalmijn! Some initial comments

  • I've tried testing it with a local dataset in a project UTM CRS but than the dres after transforming (line 2305) becomes negative and the I get an error on the next line.
  • Could you add some more comments to explain what certain blocks of code are doing, For instance I don't understand what lines 2304-2306 are for.
  • out of curiosity, would it be possible to reuse (part of) to_xyz_tiles after you have created the first zoom level (close to the native data resolution) for the subsequent zoom levels?

Let me know if something is unclear.

hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
@dalmijn
Copy link
Contributor Author

dalmijn commented Jul 14, 2023

@DirkEilander Your last point (copying from 'to_xyz_tiles') is one I thought of for a bit when I can coding. However I really need to have the locations of the tiles (windows basically) as they are fixed. That meant I could not translate the code from 'to_xyz_tiles' one to one, where this is not used whatsoever. Maybe talking in person I can explain more clearly.

@dalmijn
Copy link
Contributor Author

dalmijn commented Jul 15, 2023

No clue why Sonarcloud failed where it did...

Copy link
Contributor

@savente93 savente93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't totally attest to the semantics of the code, but assuming that's okay I'd say this looks okay. I left a few comments regarding code quality and possibilities for optimizing memory if that is an issue. Should still wait for Dirks approval to make sure the calculations are actually correct.

hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
hydromt/raster.py Outdated Show resolved Hide resolved
@dalmijn
Copy link
Contributor Author

dalmijn commented Sep 14, 2023

I processed the comments, reshuffled stuff a bit and made it more readable (I think).
Added a new method to_webviewer_tiles.
So It now also fixes: #488

Copy link
Contributor

@savente93 savente93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel I can say anything meaningful about the maths without taking a significant amount of time to understand it. The code looks good, with the comment that the mercantile package seems to not be actively maintained. if it is mostly because it's considered stable, (which I don't feel qualified to comment on) then it's okay, but otherwise it might be good if we can find a suitably maintained alternative. Other than that LGTM but I'll leave the final approval for Dirk

@dalmijn
Copy link
Contributor Author

dalmijn commented Sep 19, 2023

@savente93 It's designated as a stable release. I'd say it is indeed considered stable and just finished.

Copy link
Contributor

@DirkEilander DirkEilander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the last updates @dalmijn! A few very small final changes left before merging. Can you apply these and merge?

Comment on lines 2356 to 2358
# Setting up information for zoomlevel calculation and
# determination of tile windows
# This section is purely for the resolution
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this section be moved inside the if max_lvl is None: statement?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -2292,6 +2292,13 @@ def to_slippy_tiles(
If None, the zoomlevels will be determined based on the data resolution
driver : str, optional
file output driver, one of 'png', 'netcdf4' or 'GTiff'
cmap : str | object, optional
A colormap, either defined by a string and imported from matplotlib
via that string or as a ListedColormap object from matplotlib itself.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
via that string or as a ListedColormap object from matplotlib itself.
via that string or as a Colormap object from matplotlib itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The object is called a ListedColormap, but I will change it.

Comment on lines 2359 to 2371
bounds_4326_clip = list(obj.raster.transform_bounds("EPSG:4326"))
bounds_4326_clip[:2] = map(
max,
zip(bounds_4326_clip[:2], (-180, -y_ext)),
)
bounds_4326_clip[2:] = map(
min,
zip(bounds_4326_clip[2:], (180, y_ext)),
)
obj_clipped_to_pseudo = obj.raster.clip_bbox(
bounds_4326_clip,
crs="EPSG:4326",
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this do the same? Or am I missing something?

Suggested change
bounds_4326_clip = list(obj.raster.transform_bounds("EPSG:4326"))
bounds_4326_clip[:2] = map(
max,
zip(bounds_4326_clip[:2], (-180, -y_ext)),
)
bounds_4326_clip[2:] = map(
min,
zip(bounds_4326_clip[2:], (180, y_ext)),
)
obj_clipped_to_pseudo = obj.raster.clip_bbox(
bounds_4326_clip,
crs="EPSG:4326",
)
obj_clipped_to_pseudo = obj.raster.clip_bbox(
[-180, -y_ext, 180, y_ext],
crs="EPSG:4326",
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I just wrote it way too explicitly..., kind of laughed when I saw this.

@dalmijn
Copy link
Contributor Author

dalmijn commented Sep 22, 2023

Done all the last stuff, so I'll merge it.

@dalmijn dalmijn merged commit 8280221 into main Sep 22, 2023
8 checks passed
@savente93 savente93 deleted the osm branch September 22, 2023 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: OSM tiling (memory usage)
4 participants