Garbled data in the tags field ('name' tags with accents).

[mboeringa]

What version of osm2pgsql are you using?

1.7.2 (1.7.0-236-g8b9eedc9)

What operating system and PostgreSQL/PostGIS version are you using?

Ubuntu 22.01.4 LTS with latest updates. PostgreSQL 15.1

Tell us something about your system

What did you do exactly?

Import Italy extract of Geofabrik

What did you expect to happen?

Data is properly imported

What did happen instead?

Data in the 'tags' field is garbled. Special characters do not come over properly, e.g. https://www.openstreetmap.org/node/3225272477#map=19/46.05451/9.43078
is an example node, in this area of the Italian Alps there are many more.

What did you do to try analyzing the problem?

Looked in the 'planet_osm_point' table:

[joto]

The question was "What did you do exactly?". The answer "Import Italy extract of Geofabrik" doesn't answer that. What command line did you use?

Also: Is your database set to UTF8? (see https://osm2pgsql.org/doc/manual.html#creating-a-database )

[mboeringa]

Same as always:
osm2pgsql -O flex -d <YOUR_DATABASE> -U <YOUR_USERNAME> --tablespace-slim-data <YOUR_TABLESPACE> --tablespace-slim-index <YOUR_TABLESPACE> --create -W --slim --drop -C 40000 -P 5433 -S 'osm2pgsql/flex-config/openstreetmap-carto.lua' 'data/italy-latest.osm.pbf'

And yes, pgAdmin defaults to UTF8 for creating any new database, and this one is UTF8.

I now compiled from the latest master, and it still shows the same issue.

[mboeringa]

'tags' column is defined as JSONB by the way, using the new flex output options for that.

[lonvia]

That is the essential information you should have reported to begin with.

Minimal example to reproduce:

echo 'n1 Tfoo=bar,name=höchst x1 y1' | ./osm2pgsql -d gis --style ../flex-config/simple.lua -O flex - -r opl

ends up with:

me@machine:~/osm2pgsql/build$ psql gis
psql (13.8 (Debian 13.8-0+deb11u1))
Type "help" for help.

gis=# select * from pois;
 node_id |               tags                |                        geom                        
---------+-----------------------------------+----------------------------------------------------
       1 | {"foo": "bar", "name": "höchst"} | 0101000020110F0000310C4ADA772DFB406BF12149D22DFB40
(1 row)

gis=#

[lonvia]

Bisecting identifies not surprisingly 059f5d5 as the culprit.

[mboeringa]

Maybe this OpenStreetMap Diary entry with posts by prominent OpenStreetMap members like Andy Allen, Simon Poole and mmd also provides some food for thought:

OpenStreetMap Isn't Unicode

The case of "surrogate pairs" Andy mentions was certainly interesting to me having little background in this specific matter...

[lonvia]

Fixed with #1896.