Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on Linux, nix-installed hledger won't handle non-ascii data, even with locale set #1033

Open
simonmichael opened this issue May 31, 2019 · 7 comments

Comments

@simonmichael
Copy link
Owner

@simonmichael simonmichael commented May 31, 2019

Reported by @vifon: hledger installed via the nix command on the download page fails if the journal contains non-ascii characters. Reproduced on Gentoo and Ubuntu GNU/Linux; does not happen on mac.

Eg:

$ nix-env -i -f https://github.com/NixOS/nixpkgs/archive/fe41fd.tar.gz -A hledger; curl -s https://raw.githubusercontent.com/simonmichael/hledger/master/examples/unicode.journal | ~/.nix-profile/bin/hledger -f- print
...
hledger: <stdin>: hGetContents: invalid argument (invalid byte sequence)
$ echo $LANG
en_US.utf8
$ locale -a | grep US
en_US.iso885915
en_US.utf8
$ curl -s https://raw.githubusercontent.com/simonmichael/hledger/master/examples/unicode.journal | ~/.local/bin/hledger -f- print
(works)
@simonmichael simonmichael changed the title nix-installed hledger won't handle non-ascii data, even with locale set on Linux, nix-installed hledger won't handle non-ascii data, even with locale set May 31, 2019
simonmichael added a commit that referenced this issue May 31, 2019
@simonmichael
Copy link
Owner Author

@simonmichael simonmichael commented May 31, 2019

@peti, any idea what might be the cause ?

@peti
Copy link
Contributor

@peti peti commented Jun 1, 2019

The Nix glibc does not know any locales (other than C). Locale data resides in a separate package, glibcLocales, and glibc can only find that package through an environment variable

LOCALE_ARCHIVE="$(nix-build --no-out-link "<nixpkgs>" -A glibcLocales)/lib/locale/locale-archive"

that needs to be set at run-time. Alternatively, that variable can also point to the locale archive of the native host OS:

LOCALE_ARCHIVE="/usr/lib/locale/locale-archive"

This separation exists so that Nix users can choose a custom set of locales they'd like to install without affecting glibc. If glibc would have a reference to the locale data, then any change to the locate data would trigger a complete re-build of the entire system.

@simonmichael
Copy link
Owner Author

@simonmichael simonmichael commented Jun 1, 2019

Wow, good info, thanks. What do you think is the best approach for reliable install commands that take this into account ? This worked on Linux:

$ export LOCALE_ARCHIVE="$(nix-build --no-out-link '<nixpkgs>' -A glibcLocales)/lib/locale/locale-archive"
$ nix-env -i -f https://github.com/NixOS/nixpkgs/archive/fe41fd.tar.gz -A hledger hledger-web hledger-ui

but not on Mac:

$ nix-build --no-out-link '<nixpkgs>' -A glibcLocales
error: expression does not evaluate to a derivation (or a set or list of those)

@simonmichael
Copy link
Owner Author

@simonmichael simonmichael commented Jun 1, 2019

Also, I'm curious why it doesn't require this extra step on Mac.

@peti
Copy link
Contributor

@peti peti commented Jun 1, 2019

Nix doesn't use glibc on Darwin. I believe Darwin builds use the native libc, so the whole "re-building the world" problem doesn't exist on that OS.

@samm81
Copy link

@samm81 samm81 commented Apr 10, 2020

just wanted to chime in and say this fixed my issue on Ubuntu 18.04! thank you 😁

@tbm
Copy link
Contributor

@tbm tbm commented Jul 26, 2020

I just ran into this issue. It would be good if the file line (and character position) would be displayed so at least users can check what hledger doesn't like about the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants