[DOCS] Add blog post: reading OSM PBF into Sedona - #3160
Merged
Conversation
Walks through reading raw .osm.pbf into a Sedona DataFrame with the osmpbf format: nodes to points, assembling way LineStrings from ordered node refs (posexplode + join), tag filtering and spheroid road-length stats, plus edit-history analysis via timestamp/version. All examples run against the bundled Monaco extract. Adds two self-contained SVG figures.
Add a section covering single-file splitting across the cluster, object-storage reads, and the refs-to-coordinates assembly as a distributed join.
New figure: one planet file split into byte-range blocks, decoded in parallel, landing as one Sedona DataFrame. Cover tagline and chip now carry the distributed-scale message.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Apache Sedona blog post that introduces the native osmpbf (OpenStreetMap PBF) reader and demonstrates turning raw OSM elements (nodes/ways/relations) into usable geometries and spatial analytics in Sedona/Spark SQL.
Changes:
- Adds a new blog post walkthrough for reading
.osm.pbfdirectly into a Sedona DataFrame and querying it with Spatial SQL. - Includes examples for node→point construction, way→LineString reconstruction via ordered node refs, and basic measurement/temporal metadata queries.
- Adds three accompanying SVG figures used by the post (cover + two explanatory diagrams).
Reviewed changes
Copilot reviewed 1 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| docs/blog/posts/osm-pbf-reader.md | New blog post documenting end-to-end .osm.pbf ingestion and analysis patterns in Sedona Spark. |
| docs/blog/posts/osm-pbf-reader-cover.svg | New cover illustration for the blog post. |
| docs/blog/posts/osm-pbf-reader-assembly.svg | Diagram explaining ordered way-ref expansion and geometry reconstruction. |
| docs/blog/posts/osm-pbf-reader-scale.svg | Diagram explaining splittable/parallel decoding and scaling characteristics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+107
to
+124
| way_pts AS ( -- one row per (way, vertex), ordered | ||
| SELECT w.id AS way_id, w.tags AS tags, w.pos AS seq, n.lon, n.lat | ||
| FROM (SELECT id, tags, posexplode(refs) AS (pos, ref) | ||
| FROM osm WHERE kind = 'way') w | ||
| JOIN nodes n ON n.id = w.ref | ||
| ) | ||
| SELECT way_id, | ||
| any_value(tags)['highway'] AS highway, | ||
| ST_GeomFromText(concat('LINESTRING(', | ||
| array_join( | ||
| transform(array_sort(collect_list(struct(seq, lon, lat))), | ||
| x -> concat(x.lon, ' ', x.lat)), | ||
| ', '), | ||
| ')')) AS geom | ||
| FROM way_pts | ||
| GROUP BY way_id | ||
| HAVING count(*) >= 2 AND any_value(tags)['highway'] IS NOT NULL | ||
| """) |
Member
Author
There was a problem hiding this comment.
Fixed in 2b156ed. The highway tag is now extracted before the join with the filter pushed ahead of it, grouping is on (way_id, highway), and the LineString is built with ST_MakeLine over the sorted point array. Verified against the Monaco extract: identical results (3334 ways, 163.9 km, same per-class breakdown).
Extract the highway tag before the join and group by (way_id, highway) instead of using the non-deterministic any_value aggregate; assemble the LineString with ST_MakeLine over the sorted point array instead of WKT concatenation. Verified against the Monaco extract: identical results (3334 ways, 163.9 km, same per-class table).
jiayuasu
added a commit
that referenced
this pull request
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[DOCS] my subjectWhat changes were proposed in this PR?
A new blog post introducing the OSM PBF reader (
osmpbfformat): reading raw OpenStreetMap.osm.pbfextracts into a Sedona DataFrame with no conversion step. The reader currently has no dedicated tutorial or blog page, so this fills a docs gap..osm.pbfextract; the raw OSM element model (nodes / ways / relations) as a single tableST_Point(POI examples)posexplode+ join + ordered rebuild, with a figure explaining the patternST_LengthSpheroidper highway class)timestamp/versionmetadata fields, with a note that public Geofabrik extracts stripuser/uid/changesetHow was this patch tested?
spark/common/src/test/resources/osmpbf/monaco-latest.osm.pbf); the console output shown in the post is the real output, so readers can reproduce it verbatim.mkdocs serveand the rendered post verified: both figures load, code blocks highlight, and the page renders under the blog index.Did this PR include necessary documentation updates?