Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on Linux, nix-installed hledger won't handle non-ascii data, even with locale set #1033

Open
simonmichael opened this issue May 31, 2019 · 5 comments

Comments

Projects
None yet
2 participants
@simonmichael
Copy link
Owner

commented May 31, 2019

Reported by @Vifon: hledger installed via the nix command on the download page fails if the journal contains non-ascii characters. Reproduced on Gentoo and Ubuntu GNU/Linux; does not happen on mac.

Eg:

$ nix-env -i -f https://github.com/NixOS/nixpkgs/archive/fe41fd.tar.gz -A hledger; curl -s https://raw.githubusercontent.com/simonmichael/hledger/master/examples/unicode.journal | ~/.nix-profile/bin/hledger -f- print
...
hledger: <stdin>: hGetContents: invalid argument (invalid byte sequence)
$ echo $LANG
en_US.utf8
$ locale -a | grep US
en_US.iso885915
en_US.utf8
$ curl -s https://raw.githubusercontent.com/simonmichael/hledger/master/examples/unicode.journal | ~/.local/bin/hledger -f- print
(works)

@simonmichael simonmichael changed the title nix-installed hledger won't handle non-ascii data, even with locale set on Linux, nix-installed hledger won't handle non-ascii data, even with locale set May 31, 2019

simonmichael added a commit that referenced this issue May 31, 2019

@simonmichael

This comment has been minimized.

Copy link
Owner Author

commented May 31, 2019

@peti, any idea what might be the cause ?

@peti

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2019

The Nix glibc does not know any locales (other than C). Locale data resides in a separate package, glibcLocales, and glibc can only find that package through an environment variable

LOCALE_ARCHIVE="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"

that needs to be set at run-time. Alternatively, that variable can also point to the locale archive of the native host OS:

LOCALE_ARCHIVE="/usr/lib/locale/locale-archive"

This separation exists so that Nix users can choose a custom set of locales they'd like to install without affecting glibc. If glibc would have a reference to the locale data, then any change to the locate data would trigger a complete re-build of the entire system.

@simonmichael

This comment has been minimized.

Copy link
Owner Author

commented Jun 1, 2019

Wow, good info, thanks. What do you think is the best approach for reliable install commands that take this into account ? This worked on Linux:

$ export LOCALE_ARCHIVE="$(nix-build --no-out-link '<nixpkgs>' -A glibcLocales)/lib/locale/locale-archive"
$ nix-env -i -f https://github.com/NixOS/nixpkgs/archive/fe41fd.tar.gz -A hledger hledger-web hledger-ui

but not on Mac:

$ nix-build --no-out-link '<nixpkgs>' -A glibcLocales
error: expression does not evaluate to a derivation (or a set or list of those)
@simonmichael

This comment has been minimized.

Copy link
Owner Author

commented Jun 1, 2019

Also, I'm curious why it doesn't require this extra step on Mac.

@peti

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2019

Nix doesn't use glibc on Darwin. I believe Darwin builds use the native libc, so the whole "re-building the world" problem doesn't exist on that OS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.