Unable to read MBTiles in paths containing multibytes

[kohei-hasegawa]

What is the bug or the crash?

Unavailable layer!
Layer data source could not be found.
Click to set a new data source

"テスト" in the layer or file name is Japanese and means test in English.

Error in Layer List

Repair Data Source screen

Steps to reproduce the issue

1. Save any MBTiles file to a path containing multibytes
   example: "C:\OSGeo4W\テスト.mbtiles", "C:\OSGeo4W\テスト\test.mbtiles", "C:\OSGeo4W\テスト\テスト.mbtiles"
2. Drag and drop MBTiles files into QGIS
3. "Unavailable layer! Layer data source could not be found. Click to set a new data source" is displayed on the layer.

Versions

QGIS version
3.34.3-Prizren
QGIS code revision
4737323
Qt version
5.15.3
Python version
3.9.18
GDAL/OGR version
3.8.3
PROJ version
9.3.1
EPSG Registry database version
v10.098 (2023-11-24)
GEOS version
3.12.1-CAPI-1.18.1
SQLite version
3.41.1
PDAL version
2.6.0
PostgreSQL client version
15.2
SpatiaLite version
5.1.0
QWT version
6.1.6
QScintilla2 version
2.13.4
OS version
Windows 10 Version 2009

Active Python plugins
QuickOSM
2.2.3
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.6
processing
2.12.99

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

To try out the new profile, I started QGIS in PowerShell with the following command
PS C:\OSGeo4W\bin> .\qgis-bin.exe --profile english --lang en-us

[agiudiceandrea]

@kohei-hasegawa thanks for reporting. Does the issue only occur trying to import an MBTiles layer file from a path containing Japanese characters or also trying to import any other layer file from a path containing Japanese characters?

[kohei-hasegawa]

thanks for reporting. Does the issue only occur trying to import an MBTiles layer file from a path containing containing Japanese characters or also trying to import any other layer file from a path containing Japanese characters?

The same problem exists no matter which Japanese language is used.
This problem was reported on another website on March 18, 2022.
https://groups.google.com/g/qgisshitumon01/c/B9EWb0RCvvY

The 2022 report also occurred in "航空写真" and in Chinese characters meaning aerial photography in English.

As far as I know, the problem occurs when any of the Hiragana, Katakana, or Kanji characters commonly used in Japan are included.
Presumably, the same will occur in Chinese and Korean.

[agiudiceandrea]

@kohei-hasegawa, does the issue only occur trying to import an MBTiles layer file or does it also occur trying to import any other layer file (e.g. ESRI Shapefile, or GeoPackage, or GeoTIFF, ...)?
Does the issue only occurs trying to import the layer via drag&drop or does it also occur using the Browser panel or the Data Source Manager?

[kohei-hasegawa]

@kohei-hasegawa, does the issue only occur trying to import an MBTiles layer file or does it also occur trying to import any other layer file (e.g. ESRI Shapefile, or GeoPackage, or GeoTIFF, ...)? Does the issue only occurs trying to import the layer via drag&drop or does it also occur using the Browser panel or the Data Source Manager?

@agiudiceandrea
Thanks for gathering more information.

I have worked with Esri Shape files, GeoPackages, GeoJSON, GeoTIFF, and many other formats of data, but the problem occurs only with MBTiles.

The same problem occurs in the Browser panel and Data Source Manager.

The error displayed in Data Source Manager showed URI encoding.
Perhaps the encoding is causing the path to be unrecognizable.

Error Description
Invalid Data Source: type=mbtiles&url=C:%5CUsers%5Ckohei%5CDownloads%5C%E3%83%86%E3%82%B9%E3%83%88.mbtiles is not a valid or recognized data source, error: ""

Paths set in the UI
C:\Users\kohei\Downloads\テスト.mbtiles

I have attached the MBTiles used in this test.
The contents are the zoom levels 0-4 of the XYZ tiles published by the Geospatial Information Authority of Japan.
テスト.zip

[agiudiceandrea]

@kohei-hasegawa, thanks for checking. Anyway the provided layer is not a "Vector Tile" layer, but it is a raster layer: please try if it is possible to add it using the Data Source Manager -> Raster tab.

It seems to me the issue only occurs importing the MBTiles layer file having name テスト.mbtiles via drag&drop or using the Browser panel, while it doesn't occur using the Data Source Manager -> Raster tab.

The issue occurs also with various characters other than Japanese or Chinese or Korean in the path: for example ŠŽšžŁČĚŐŰ i.e. characters after unicode U+00FF.

Additional notes:

• when an MBTiles layer (either containing or non containing characters after unicode U+00FF in the path) is added to the project using Data Source Manager -> Raster, then the MBTiles layer is handled by the gdal provider and the issue doesn't occur;
  

• when an MBTiles layer (either containing or non containing characters after unicode U+00FF in the path) is added to the project via drag&drop or using the Browser panel, then the MBTiles layer is handled by the wms provider and

  • the issue occurs if the MBTiles layer's path contains characters after unicode U+00FF
    

  • while it doesn't occur if the MBTiles layer's path doesn't contains characters after unicode U+00FF
    

[agiudiceandrea]

Testing some LTR versions, it looks like:

• the issue doesn't occur using QGIS 3.10, since with such version the MBTiles layers are always handled by the gdal provider.

• the issue partially doesn't occur using QGIS 3.16: if the MBTiles layer's path contains multi-bytes characters and it is imported via drag&drop or using the Browser panel, then the wms provider fails with the error Invalid Layer : WMS provider Cannot open MBTiles database Raster layer Provider is not valid (provider: wms, URI: type=mbtiles&url=file:///F:/QGIS/[...]/JAP/テスト.mbtiles and then the MBTiles layer is automatically handled by the gdal provider and imported as gdal raster layer.

• the issue occurs using QGIS 3.22 and next versions QGIS 3.28, QGIS 3.34.

[agiudiceandrea]

Hi @wonder-sk, have you seen this issue report about MBTiles raster layers in paths containing multibytes that are not correctly handled by the WMS provider?

[kohei-hasegawa]

@agiudiceandrea

Thanks for your help.
I added the layer as a raster as instructed and it loaded correctly.

Please correct me if I am wrong in my perception.

• Problems with loading files when dragging and dropping
• The problem occurs with multibyte characters (U+00FF or later).
• There is a problem with the way the provider loads the files in the first place.
  MBTiles (raster) incorrect behaviour when added from Data Source Manager > Raster #36312

Do you need any additional information from me?

[nirvn]

@kohei-hasegawa , thanks for the report, this was a rather serious limitation/issue I'm glad we could address. The fix should make its way to the next public release (end of February).