encoding CP852 missing in windows instalation

[AlzbetaGardonova]

**There is no possibility to set encoding CP 852 to layer in windows instalation. **
Code CP 852 is not in the list of encodings. But is available in ubuntu instalation. If I have shapefile with this encoding defined in cpg file the encoding pre-set to System. QGIS seems like ignore cpg file. There is no more possibility so set this by settings ->datasource->ignore shapefile encoding declaration in this version.

QGIS and OS versions
QGIS 3.10.6 on win 10 and ubuntu 18.04

[nyalldawson]

This setting was moved to Layer Properties - Source tab.

[kadarivan]

@nyalldawson The character encoding settings was moved, but CP852 still missing on Win10.

In previous QGIS versions, OGR decoded CP852 correctly with the "ignore shapefile encoding declaration" option. Without this option QGIS can't decode CP852 on Win10.

[agiudiceandrea]

hi @AlzbetaGardonova @kadarivan, it seems the QGIS encodings list is taken from the Qt codecs list, and CP852 seems missing in current Qt version (5.11.2) used by OSGeo4W at least on Windows with Italian locale.
Anyway, if your system is in CP852, shouldn't "System" mean CP852?

[kadarivan]

CP852/IBM852 is a very old codepage from the DOS era. Just some legacy program export data in this codepage (like Hungarian Public Road Company's databank). No one use it as system default encoding.
Before a recent patch OGR converted on-the-fly every data to utf8, so the user don't have to select codepage manually. But now this option is missing so qgis not support CP852 (and lots of other codepages I guess).

[agiudiceandrea]

hi @nyalldawson, I did some testing and it seems to me that actually it's impossible to properly read a CP852 encoded shapefile using QGIS 3.10.6 on Windows, while it is possible using QGIS 3.10.1.

This test_852.zip is a zipped point shapefile, test_852.shp, with a single text field, "test852", containing a single record with the following 4 char in hex: 0x61, 0x62, 0x63, 0x8A, as the field value.

The test_852.cpg file contains the string "852".

In the Code page 852 the hex chars 0x61, 0x62, 0x63, 0x8A should be decoded as abcŐ (the last char is LATIN CAPITAL LETTER O WITH DOUBLE ACUTE).

On Windows 7 64 bit (Italian language),

• using QGIS 3.10.1/GDAL 3.0.2 (with "Ignore shapefile encoding declaration" set to off, as default), the chars are correctly decoded using IBM/CP 852:
  
  the Layer properties display Data source encoding: UTF-8, grayed out
  

• using QGIS 3.10.6/GDAL 3.0.4 (there is no "Ignore shapefile encoding declaration" setting), the chars are decoded incorrectly using Windows-1252 code page (which is the System code page for Italian language Windows) as abcŠ (the last char is LATIN CAPITAL LETTER S WITH CARON):
  
  the Layer properties display Data source encoding: System
  
  It's possible to manually change the code page, e.g. with code page CP850
  
  but it is not possible to obtain the correct decoding, since the encoding list lacks of IBM/CP852 codec as the QGIS encodings list is taken from the Qt codecs list, and CP852 seems missing in current Qt version (5.11.2) used by OSGeo4W.

So it seems this regression may have be introduced in the effort of fixing other shapefile encoding issues like with PR #34381 / #34607

[agiudiceandrea]

so qgis not support CP852 (and lots of other codepages I guess).

You are right @kadarivan,
on Windows 7 / 10, QgsVectorDataProvider.availableEncodings() outputs 135 available codecs names using QGIS 3.4 / 3.10 / 3.12 running against Qt 5.11.2, and 134 available codecs names using QGIS 2.18.23 running against Qt 4.8.5,
while on Ubuntu 20.04, QgsVectorDataProvider.availableEncodings() outputs 812 available codecs names using QGIS 3.12 running against Qt 5.12.8.