Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

locale and LC_ALL on NetBSD #11550

Closed
p5pRT opened this issue Aug 4, 2011 · 10 comments
Closed

locale and LC_ALL on NetBSD #11550

p5pRT opened this issue Aug 4, 2011 · 10 comments

Comments

@p5pRT
Copy link

@p5pRT p5pRT commented Aug 4, 2011

Migrated from rt.perl.org#96240 (status was 'resolved')

Searchable as RT96240$

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 4, 2011

From msporleder@gmail.com

Created by msporleder@gmail.com

Setting LC_ALL causes the locale warning. My system has locale files
and I see perl reading them. How can I troubleshoot this more?

d_setlocale='define';

~ $ export LC_ALL=fr_FR.ISO8859-1~ $ perl -e '1;'
perl​: warning​: Setting locale failed.
perl​: warning​: Please check that your locale settings​:
  LC_ALL = "fr_FR.ISO8859-1",
  LANG = (unset)
  are supported and installed on your system.
perl​: warning​: Falling back to the standard locale ("C").

~ $ locale -a|grep fr
fr
fr_BE.ISO8859-1
fr_BE.ISO8859-15
fr_BE.UTF-8
fr_CA.ISO8859-1
fr_CA.ISO8859-15
fr_CA.UTF-8
fr_CH.ISO8859-1
fr_CH.ISO8859-15
fr_CH.UTF-8
fr_FR.ISO8859-1
fr_FR.ISO8859-15
fr_FR.UTF-8

22799 1 perl CALL open(0x7f7fffffc470,0,0x1b6)
22799 1 perl NAMI "/usr/share/locale/fr_FR.ISO8859-1/LC_CTYPE"
22799 1 perl RET open 3
22799 1 perl CALL __fstat30(3,0x7f7fffffb700)
22799 1 perl RET __fstat30 0
22799 1 perl CALL __fstat30(3,0x7f7fffffb4f0)
22799 1 perl RET __fstat30 0
22799 1 perl CALL lseek(3,0,0,1)
22799 1 perl RET lseek 0
22799 1 perl CALL lseek(3,0,0,0)
22799 1 perl RET lseek 0
22799 1 perl CALL read(3,0x7f7ffd219000,0x4000)
22799 1 perl GIO fd 3 read 3241 bytes
  "RuneCT10NONE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\M^?\
  \M^?\M^?\M-}\0\0\0\0\0\0\^B\0\0\0\^B\0\0\0\^B\0\0\0\^B\0\0\0\^B\0\0\0\^B\0\0\0\^B\0
<snip>
  \0\0\0\0\0\0\0\0\0)\0\0\0\0Latin-1 character set, CODESET=ISO8859-1\0"
22799 1 perl RET read 3241/0xca9
22799 1 perl CALL close(3)
22799 1 perl RET close 0
22799 1 perl CALL open(0x7f7fffffc470,0,0)
22799 1 perl NAMI "/usr/share/locale/fr_FR.ISO8859-1/LC_MONETARY"
22799 1 perl RET open 3
22799 1 perl CALL fcntl(3,2,1)

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.14.1:

Configured by msporled at Wed Aug  3 07:06:43 EDT 2011.

Summary of my perl5 (revision 5 version 14 subversion 1) configuration:
   
  Platform:
    osname=netbsd, osvers=5.1, archname=amd64-netbsd
    uname='netbsd vc136-15.vc.panix.com 5.1 netbsd 5.1 (panix-vc) #0: thu mar 10 01:49:14 est 2011 root@juggler.panix.com:miscobjmiscdevelnetbsd5.1srcsysarchamd64compilepanix-vc amd64 '
    config_args='-de -Dprefix=/var/tmp/perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/pkg/include',
    optimize='-O',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/pkg/include'
    ccversion='', gccversion='4.1.3 20080704 prerelease (NetBSD nb2 20081120)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -Wl,-rpath,/usr/pkg/lib -Wl,-rpath,/usr/local/lib -fstack-protector -L/usr/pkg/lib'
    libpth=/usr/pkg/lib /lib /usr/lib
    libs=-lgdbm -lm -lcrypt -lutil -lc -lposix
    perllibs=-lm -lcrypt -lutil -lc -lposix
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E '
    cccdlflags='-DPIC -fPIC ', lddlflags='--whole-archive -shared  -L/usr/pkg/lib -fstack-protector'

Locally applied patches:
    


@INC for perl 5.14.1:
    /var/tmp/perl/lib/site_perl/5.14.1/amd64-netbsd
    /var/tmp/perl/lib/site_perl/5.14.1
    /var/tmp/perl/lib/5.14.1/amd64-netbsd
    /var/tmp/perl/lib/5.14.1
    .


Environment for perl 5.14.1:
    HOME=/home/msporled
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/msporled/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/games:/usr/local/bin:/usr/local/sbin
    PERL_BADLANG (unset)
    SHELL=/usr/pkg/bin/bash

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 4, 2011

From @tonycoz

On Wed, Aug 03, 2011 at 06​:54​:50PM -0700, msporleder@​gmail.com (via RT) wrote​:

Setting LC_ALL causes the locale warning. My system has locale files
and I see perl reading them. How can I troubleshoot this more?

d_setlocale='define';

~ $ export LC_ALL=fr_FR.ISO8859-1~ $ perl -e '1;'
perl​: warning​: Setting locale failed.
perl​: warning​: Please check that your locale settings​:
LC_ALL = "fr_FR.ISO8859-1",
LANG = (unset)
are supported and installed on your system.
perl​: warning​: Falling back to the standard locale ("C").

From what I can see in reading locale.c, perl attempts to setlocale()
for each category. On NetBSD this fails for LC_COLLATE​:

deimos$ uname -prs
NetBSD 5.1 x86_64
deimos$ LC_ALL=fr_FR.ISO8859-1 ./setlocale
LC_ALL : C/fr_FR.ISO8859-1/fr_FR.ISO8859-1/fr_FR.ISO8859-1/fr_FR.ISO8859-1/fr_FR.ISO8859-1
LC_COLLATE : (null)
LC_CTYPE : fr_FR.ISO8859-1
LC_MESSAGES : fr_FR.ISO8859-1
LC_MONETARY : fr_FR.ISO8859-1
LC_NUMERIC : fr_FR.ISO8859-1
LC_TIME : fr_FR.ISO8859-1

Setting only the categories you need works (as long as they aren't
LC_COLLATE)​:

deimos$ LC_NUMERIC=fr_FR.ISO8859-1 LC_CTYPE=fr_FR.ISO8859-1 ./setlocale
LC_ALL : C/fr_FR.ISO8859-1/C/fr_FR.ISO8859-1/C/C
LC_COLLATE : C
LC_CTYPE : fr_FR.ISO8859-1
LC_MESSAGES : C
LC_MONETARY : C
LC_NUMERIC : fr_FR.ISO8859-1
LC_TIME : C
deimos$ LC_COLLATE=fr_FR.ISO8859-1 ./setlocale
LC_ALL : C
LC_COLLATE : (null)
LC_CTYPE : C
LC_MESSAGES : C
LC_MONETARY : C
LC_NUMERIC : C
LC_TIME : C

The source for the setlocale tool​:

deimos$ cat setlocale.c
#include <locale.h>
#include <stdio.h>
#include <errno.h>

#define entry(s) { s, #s }
#define quote_(s) #s
#define quote(s) quote_(s)

struct locale_category {
  int id;
  char const *name;
} categories[] = {
  entry(LC_ALL),
  entry(LC_COLLATE),
  entry(LC_CTYPE),
  entry(LC_MESSAGES),
  entry(LC_MONETARY),
  entry(LC_NUMERIC),
  entry(LC_TIME)
};

const int category_count = sizeof(categories) / sizeof(*categories);

int main() {
  int i;
  for (i = 0; i < category_count; ++i)
  printf("%-12s​: %s\n", categories[i].name, setlocale(categories[i].id, ""));

  return 0;
}

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 4, 2011

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jul 8, 2013

From @khwilliamson

On Thu Aug 04 03​:57​:22 2011, tonyc wrote​:

On Wed, Aug 03, 2011 at 06​:54​:50PM -0700, msporleder@​gmail.com (via
RT) wrote​:

Setting LC_ALL causes the locale warning. My system has locale
files
and I see perl reading them. How can I troubleshoot this more?

d_setlocale='define';

~ $ export LC_ALL=fr_FR.ISO8859-1~ $ perl -e '1;'
perl​: warning​: Setting locale failed.
perl​: warning​: Please check that your locale settings​:
LC_ALL = "fr_FR.ISO8859-1",
LANG = (unset)
are supported and installed on your system.
perl​: warning​: Falling back to the standard locale ("C").

From what I can see in reading locale.c, perl attempts to setlocale()
for each category. On NetBSD this fails for LC_COLLATE​:

deimos$ uname -prs
NetBSD 5.1 x86_64
deimos$ LC_ALL=fr_FR.ISO8859-1 ./setlocale
LC_ALL : C/fr_FR.ISO8859-1/fr_FR.ISO8859-1/fr_FR.ISO8859-
1/fr_FR.ISO8859-1/fr_FR.ISO8859-1
LC_COLLATE : (null)
LC_CTYPE : fr_FR.ISO8859-1
LC_MESSAGES : fr_FR.ISO8859-1
LC_MONETARY : fr_FR.ISO8859-1
LC_NUMERIC : fr_FR.ISO8859-1
LC_TIME : fr_FR.ISO8859-1

Setting only the categories you need works (as long as they aren't
LC_COLLATE)​:

deimos$ LC_NUMERIC=fr_FR.ISO8859-1 LC_CTYPE=fr_FR.ISO8859-1
./setlocale
LC_ALL : C/fr_FR.ISO8859-1/C/fr_FR.ISO8859-1/C/C
LC_COLLATE : C
LC_CTYPE : fr_FR.ISO8859-1
LC_MESSAGES : C
LC_MONETARY : C
LC_NUMERIC : fr_FR.ISO8859-1
LC_TIME : C
deimos$ LC_COLLATE=fr_FR.ISO8859-1 ./setlocale
LC_ALL : C
LC_COLLATE : (null)
LC_CTYPE : C
LC_MESSAGES : C
LC_MONETARY : C
LC_NUMERIC : C
LC_TIME : C

The source for the setlocale tool​:

deimos$ cat setlocale.c
#include <locale.h>
#include <stdio.h>
#include <errno.h>

#define entry(s) { s, #s }
#define quote_(s) #s
#define quote(s) quote_(s)

struct locale_category {
int id;
char const *name;
} categories[] = {
entry(LC_ALL),
entry(LC_COLLATE),
entry(LC_CTYPE),
entry(LC_MESSAGES),
entry(LC_MONETARY),
entry(LC_NUMERIC),
entry(LC_TIME)
};

const int category_count = sizeof(categories) / sizeof(*categories);

int main() {
int i;
for (i = 0; i < category_count; ++i)
printf("%-12s​: %s\n", categories[i].name,
setlocale(categories[i].id, ""));

return 0;
}

We can't fix the underlying OS. But there is an undocumented compile
option you can use to get Perl to not try to set LC_COLLATE. It is
-DNO_LOCALE_COLLATE
This is a C flag, and can be placed into Configure using this parameter
./Configure -Accflags=-DNO_LOCALE_COLLATE ...

You can similarly turn off any other individual category.

I intend to close this ticket in 30 days unless I hear something to the
contrary
--
Karl Williamson

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Dec 10, 2013

From @khwilliamson

This is not a bug in Perl, given the C test program that exhibits similar results
I'm closing as per my statement above

--
Karl Williamson

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Dec 10, 2013

@khwilliamson - Status changed from 'open' to 'rejected'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 20, 2014

From @khwilliamson

I'm reopening this ticket because blead now has a fix for this
--
Karl Williamson

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 20, 2014

@khwilliamson - Status changed from 'rejected' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 20, 2014

From @khwilliamson

NetBSD doesn't appear to deal with LC_COLLATE, but it does in fact try to make it appear as if it does, after a fashion. If you try to set LC_COLLATE, it will fail (it might succeed if you set it to POSIX, I haven't tried). But if you set LC_ALL to something, it succeeds and returns that LC_COLLAtE is POSIX.

The Perl initialization code used to always try to set LC_COLLATE, but it now doesn't try if setting LC_ALL succeeded. This means that it will not generally fail on initialization on NetBSD.

The commit that did this is 7cd8b56


Karl Williamson

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 20, 2014

@khwilliamson - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant