Sextante output do not have the same Encoding as the inputs #15595

qgib · 2012-09-03T00:36:01Z

Author Name: Filipe Dias (@fsdias)
Original Redmine Issue: 6299
Affected QGIS version: 2.4.0
Redmine category:processing/core
Assignee: Victor Olaya

If I select as an input a layer with special characters such as "´", "~", "`" etc the output of any Sextante tool replaces these characters with weird symbols

Related issue(s): #19488 (duplicates)
Redmine related issue(s): 11174

qgib · 2012-10-19T07:22:26Z

Author Name: Filipe Dias (@fsdias)

Im changing the priority because this "corrupts" the output's table.

priority_id was changed from Normal to High

qgib · 2012-10-30T04:26:57Z

Author Name: Alexander Bruy (@alexbruy)

Not sure if I understand problem correctly. Your input file has some non-ASCII characters in attributes and in output file, produced by SEXTANTE this characters are corrupted. Right?

In this case this is not SEXTANTE issue but QGIS and GDAL one, there are number of tickets about this problem (see #14989, #15720 and related tickets). If you use master branch then check "Ignore shapefile encoding" checkbox under Settings-Options-General, restart QGIS and try again.

status_id was changed from Open to Feedback

qgib · 2012-11-01T11:02:39Z

Author Name: Filipe Dias (@fsdias)

Yes that was it.

I tried with today's QGIS Master and this no longer happened. It wasn't even necessary to check "ignore shapefile encoding".

Sinc this is/was a QGIS issue Im closing this.

status_id was changed from Feedback to Closed

qgib · 2012-11-04T09:09:24Z

Author Name: Pedro Venâncio (Pedro Venâncio)

Hi,

I'm reopening this ticket because I think that there is even a problem.

I leave here [1] a table with the behavior I get with encoding ISO-8859-1 or UTF-8, in different situations and with the same tools from QGIS Vector menu, from Sextante -> fTools or from Sextante -> SAGA.

The data I am testing is this one (originally in ISO-8859-1) [2].

Could someone confirm? I'm using Xubuntu and QGIS master.

Thanks!

[1] https://dl.dropbox.com/u/5772257/qgis/qgis_sextante_encoding_problem.xls
[2] http://www.igeo.pt/produtos/cadastro/caop/download/CAOP20121_Shapes/Cont_AAd_CAOP20121.zip

status_id was changed from Closed to Reopened

qgib · 2012-11-04T23:58:31Z

Author Name: Filipe Dias (@fsdias)

Pedro, to make sure I understand correctly (and to make sure others do to):

You made a table in which you record your findings regarding encoding problems. You used different tools and different encoding systems to test this issue. With some combinations of "tool x encoding" you found problems. But we others you didnt, is that it?

I only tested with ftools dissolve and got no error. But after reading your findings, Im lead to believe that there is a "larger issue" behind this.

Did you try Alexander Bruy's recomendation above?

qgib · 2012-11-05T01:00:09Z

Author Name: Pedro Venâncio (Pedro Venâncio)

Hi Filipe,

Exactly, and in addition, following Alexander's indication, I did the tests with the option "Ignore shapefile encoding" selected (first 4 columns in the spreadsheet attached) and not selected (last 4 columns).

The column "Result encoding" shows the result "Correct" when execution of the operation leads to a shapefile with the correct encoding, and "Incorrect" when the encoding of the resulting shapefile is wrong and different from the input layer.

qgib · 2013-04-30T00:54:13Z

Author Name: Filipe Dias (@fsdias)

I understand there have been some changes regarding Encoding in QGis Master. Is this still happening?

qgib · 2013-06-04T06:19:39Z

Author Name: Filipe Dias (@fsdias)

This is still happening and it's probably the biggest issue on Sextante right now.

priority_id was changed from High to Severe/Regression

qgib · 2013-08-23T08:40:26Z

Author Name: Borys Jurgiel (@borysiasty)

Although QGIS works properly with and without "Ignore Shapefile encoding" now, please remember there is one unresolvable problem as long as it's turned off (so GDAL conversion is not deactivated):

Since GDAL converts the source encoding to UTF-8 (more or less properly, depending on the declaration in the SHP), it doesn't pass any information about the source encoding to QGIS, so QGIS is unable to apply it to the output file. It always gets UTF8 from GDAL provider and it's a real pain for any SHP processing...

qgib · 2013-08-28T02:41:24Z

Author Name: Giovanni Manghi (@gioman)

This seems to be platform dependent, in fact I made a few tests now based on #15595 (comment) and on Windows I can always see the expected results (no garbled attributes) while on Linux I confirm the issue.

Unfortunately for us Linux users I don't think that at this stage this can be tagged as blocker.

priority_id was changed from Severe/Regression to High

qgib · 2013-08-28T03:12:54Z

Author Name: Giovanni Manghi (@gioman)

Giovanni Manghi wrote:

This seems to be platform dependent, in fact I made a few tests now based on #15595 (comment) and on Windows I can always see the expected results (no garbled attributes) while on Linux I confirm the issue.

Unfortunately for us Linux users I don't think that at this stage this can be tagged as blocker.

it does not seems to depend on the gdal/ogr version. I initially tested qgis 32bit on Windows that uses gdal 1.9.2, then I tested qgis 64bit/Windows and the results are the same (ok) and it uses gdal 1.10.

qgib · 2013-08-28T04:30:49Z

Author Name: Alexander Bruy (@alexbruy)

I can't download sample file, mentioned in previous comments, but this issue not reproducible here (Linux, GDAL 1.10.0) with my own test data (contains Cyrillic and some other non-ASCII characters).

Seems the only possible way to get broken attributes is using "System" encoding when saving file.

qgib · 2013-08-28T08:22:50Z

Author Name: Giovanni Manghi (@gioman)

It seems to be also a problem not related to QGIS/SEXTANTE, in fact I can replicate issues with the enconding just using SAGA (for example) from the command line.

qgib · 2013-11-19T03:37:41Z

Author Name: Filipe Dias (@fsdias)

Alexander Bruy wrote:

Seems the only possible way to get broken attributes is using "System" encoding when saving file.

I still get messed up characters after merging (Merge shapes - SAGA 2.0.8) correctly created .shp with ISO 8859-2

qgib · 2014-02-03T01:28:07Z

Author Name: Filipe Dias (@fsdias)

priority_id was changed from High to Severe/Regression

qgib · 2014-02-16T12:59:34Z

Author Name: Giovanni Manghi (@gioman)

Filipe Dias wrote:

Alexander Bruy wrote:

Seems the only possible way to get broken attributes is using "System" encoding when saving file.

I still get messed up characters after merging (Merge shapes - SAGA 2.0.8) correctly created .shp with ISO 8859-2

For how painful this can be, I don't think that any Sextante ticket should be a blocker, because Sextante/processing is upgreadable at any moment. This will obviously be more evident when (maybe during next dev meeting) merge the processing tickets into main qgis ones.

On the other hand I still doubt that this in just a processing issue. Did you tested the same operation in native SAGA? I personally get messed attributes there too.

status_id was changed from Reopened to Feedback
priority_id was changed from Severe/Regression to High

qgib · 2014-02-17T01:10:50Z

Author Name: Filipe Dias (@fsdias)

On SAGA 2.10, Merge Shapes doesn't corrupt the encoding.

I also tried v.overlay from QGIS and it doesn't corrupt the encoding.

Can you confirm that on your system only SAGA algs corrupt the encoding?

qgib · 2014-10-04T10:08:35Z

Author Name: Giovanni Manghi (@gioman)

project_id was changed from 78 to 17
category_id removed 63
version was configured as 2.4.0
crashes_corrupts_data was configured as 0

qgib · 2014-10-04T10:09:35Z

Author Name: Giovanni Manghi (@gioman)

category_id was configured as Processing/Core

qgib · 2015-12-19T11:10:48Z

Author Name: Giovanni Manghi (@gioman)

it seems to me that this is now fixed (at least in master), please reopen of necessary.

status_id was changed from Feedback to Closed
resolution was changed from to fixed/implemented

The problem is that some providers would still issue network requests in prepareJobs() - this should be ideally avoided, because it is run in main thread - all the work should be deferred to be done in worker thread.

qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! High Priority Processing Relating to QGIS Processing framework or individual Processing algorithms labels May 24, 2019

qgib closed this as completed May 24, 2019

qgib mentioned this issue May 25, 2019

Qgis processing toolbox is breaking encoding of output shapefile #19488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sextante output do not have the same Encoding as the inputs #15595

Sextante output do not have the same Encoding as the inputs #15595

qgib commented Sep 3, 2012 •

edited

Loading

qgib commented Oct 19, 2012

qgib commented Oct 30, 2012 •

edited

Loading

qgib commented Nov 1, 2012

qgib commented Nov 4, 2012

qgib commented Nov 4, 2012

qgib commented Nov 5, 2012

qgib commented Apr 30, 2013

qgib commented Jun 4, 2013

qgib commented Aug 23, 2013

qgib commented Aug 28, 2013 •

edited

Loading

qgib commented Aug 28, 2013 •

edited

Loading

qgib commented Aug 28, 2013

qgib commented Aug 28, 2013

qgib commented Nov 19, 2013

qgib commented Feb 3, 2014

qgib commented Feb 16, 2014

qgib commented Feb 17, 2014

qgib commented Oct 4, 2014

qgib commented Oct 4, 2014

qgib commented Dec 19, 2015

Sextante output do not have the same Encoding as the inputs #15595

Sextante output do not have the same Encoding as the inputs #15595

Comments

qgib commented Sep 3, 2012 • edited Loading

qgib commented Oct 19, 2012

qgib commented Oct 30, 2012 • edited Loading

qgib commented Nov 1, 2012

qgib commented Nov 4, 2012

qgib commented Nov 4, 2012

qgib commented Nov 5, 2012

qgib commented Apr 30, 2013

qgib commented Jun 4, 2013

qgib commented Aug 23, 2013

qgib commented Aug 28, 2013 • edited Loading

qgib commented Aug 28, 2013 • edited Loading

qgib commented Aug 28, 2013

qgib commented Aug 28, 2013

qgib commented Nov 19, 2013

qgib commented Feb 3, 2014

qgib commented Feb 16, 2014

qgib commented Feb 17, 2014

qgib commented Oct 4, 2014

qgib commented Oct 4, 2014

qgib commented Dec 19, 2015

qgib commented Sep 3, 2012 •

edited

Loading

qgib commented Oct 30, 2012 •

edited

Loading

qgib commented Aug 28, 2013 •

edited

Loading

qgib commented Aug 28, 2013 •

edited

Loading