Running mapserver code at different locale may result in unexpected behaviour

[mapserver-bot]

Reporter: szekerest
Date: 2006/04/27 - 22:39
Trac URL: http://trac.osgeo.org/mapserver/ticket/1762

I had a problem with that mapserver/mapscript is working
under the locale context of the host process, but the code
is not ready to operate in all locales so far. For example
the lexer expects dot as the decimal separator and some of
the locales use comma and atof will truncate the numbers to
the integer part causing mystical errors.

As a quick fix I suggest to call setlocale to the default
"C" at the msSetup initialization function.

The best solution however would rewriting the code to be locale
independent if it is expected.

Fixing this bug may also fix
http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=1758 

Tamas

[mapserver-bot]

Author: dmorissette
Date: 2006/04/27 - 23:13

A note about this should probably be added to the MapScript 
docs (SWIG and PHP docs) since this can potentially cause side-effects 
in scripting environments. I can take care of the PHP docs once the 
change is done.

Does anyone see a better place to document this?

[mapserver-bot]

Author: unicoletti
Date: 2006/04/28 - 14:57

From man 7 locale:

       If the second argument to setlocale() is empty string, "", for the 
default  locale,  it  is  determined
       using the following steps:

       1.     If there is a non-null environment variable LC_ALL, the value of
LC_ALL is used.

       2.     If  an  environment variable with the same name as one of the
categories above exists and is non-
              null, its value is used for that category.

       3.     If there is a non-null environment variable LANG, the value of
LANG is used.

I am attaching a simple c program that proves that atof can be fully localized
so that it should not be necessary to hard code the locale to C.

[mapserver-bot]

Author: fwarmerdam
Date: 2006/04/28 - 15:30

Umberto, 

I'm not sure what the demonstration program is intended to demonstrate. 

I would add that atof() calls in MapServer itself are not the only problem. 
We also have issues with libraries called by MapServer. 

[mapserver-bot]

Author: unicoletti
Date: 2006/04/28 - 15:36

Frank,
the bug report says:

> For example
> the lexer expects dot as the decimal separator and some of
> the locales use comma and atof will truncate the numbers to
> the integer part causing mystical errors.

and I only intended to show that atof when properly invoked can deal all locales
that it knows of.

The setlocale(LC_NUMERIC,"") call will of course affect all other libraries as
would the setlocale(LC_NUMERIC,"C").

[mapserver-bot]

Author: szekerest
Date: 2006/04/28 - 23:54

Umberto,

I think we have the inverse problem now. Because the lexer expects dot as the
decimal separator atof should be "delocalized" so as to parse the numeric string 
expecting dot regardless of the current locale setting.

Furthermore the code should run properly even if the host process or some other
libraries are tampering the locale setting of the process.

(By the way how could we set the locale of the java runtime without modifying
the java code? "-Duser.language=.. -Duser.region=.." in the command line does
not seem to work)

Tamas

[mapserver-bot]

Author: unicoletti
Date: 2006/04/29 - 20:37

> I think we have the inverse problem now. Because the lexer expects dot as the
> decimal separator atof should be "delocalized" so as to parse the numeric string 
> expecting dot regardless of the current locale setting.

I understand, I have looked at the source.

> (By the way how could we set the locale of the java runtime without modifying
> the java code? "-Duser.language=.. -Duser.region=.." in the command line does
> not seem to work)

Set LANG and LC_* variables to the locale you want (for example en_US or C to
allow . as decimal separator). Please note that Java mapscript will work even
under locales like it_IT that define , as decimal separator.

[mapserver-bot]

Author: szekerest
Date: 2006/05/01 - 00:00

>Set LANG and LC_* variables to the locale you want (for example en_US or C to
>allow . as decimal separator). Please note that Java mapscript will work even
>under locales like it_IT that define , as decimal separator.

I could not get it to work. The java runtime is using the C locale regardless of
the LANG and LC_* exports ;) To make sure of it i have printed the return value
of "setlocale(LC_NUMERIC, NULL)" somewhere in the mapscript C code.
I shall continue to believe that make test for java would also fail if we can
set the locale context of java to a setting having the decimal separator other
than "."

Tamas

[mapserver-bot]

Author: unicoletti
Date: 2006/05/08 - 10:30

> I could not get it to work. The java runtime is using the C locale regardless of
> the LANG and LC_* exports ;) To make sure of it i have printed the return value
> of "setlocale(LC_NUMERIC, NULL)" somewhere in the mapscript C code.

Tamas,
when I say that Java obeys to LANG and C_* settings I mean the jvm and PURE java
classes (like java.lang.String, Double, etc), not mapscript. Java mapscript is
written in C and Java does not propagate those settings to it. The only way Java
mapscript can be made aware of LANG and/or LC_* is by calling
setlocale(LC_ALL,""); in mapscript itself.

So Java mapscript does not suffer of this problem because it is using the system
locale (in my case it_IT) while doing its work in the jvm, but will let
mapscript work in C mode (which should be the default until setlocale is called,
which is never done in mapscript AFAIK). This probably happens because Java
never calls setlocale in its own C code.

On the other hand I suppose that the Mono interpreter is calling setlocale
somewhere and thus influencing the behaviour of C# mapscript.

> I shall continue to believe that make test for java would also fail if we can
> set the locale context of java to a setting having the decimal separator other
> than "."

You are right. If, for instance, I put a call to setlocale(LC_ALL,"") in
maputil.c Java mapscript exhibits the same behaviour of C# mapscript which makes
it impossibile to load a file that has ',' as the decimal separator.

I hope I have answered your question.

Umberto

[mapserver-bot]

attachment http://trac.osgeo.org/mapserver/attachment/ticket/1762/lc_atof.c :

   Sample program to play around with atof locale support

[mapserver-bot]

This is an automated comment

This issue has been closed due to lack of activity. This doesn't mean the issue is invalid, it simply got no attention within the last year. Please reopen with missing/relevant information if still valid.

Typically, issues fall in this state for one of the following reasons:

• Hard, impossible or not enough information to reproduce
• Missing test case
• Lack of a champion with interest and/or funding to address the issue