New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

commitBuffer: invalid argument (invalid character) #793

Closed
ThibaudDauce opened this Issue Aug 14, 2015 · 20 comments

Comments

Projects
None yet
8 participants
@ThibaudDauce
Contributor

ThibaudDauce commented Aug 14, 2015

Hi,

When I'm running stack build on my Debian 8 (for a Hakyll project), I receive this error after a few packages installed:

regex-tdfa-1.2.0: configure
zlib-0.5.4.2: download
regex-tdfa-1.2.0: build
transformers-base-0.4.4: configure
transformers-base-0.4.4: build
texmath-0.8.2: configure
texmath-0.8.2: build
zlib-0.5.4.2: configure
zlib-0.5.4.2: build
zlib-0.5.4.2: install
Progress: 4/27
--  While building package regex-tdfa-1.2.0 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure (-9)
    Logs have been written to: /var/www/.stack-work/logs/regex-tdfa-1.2.0.log

    Configuring regex-tdfa-1.2.0...
    Building regex-tdfa-1.2.0...
    Preprocessing library regex-tdfa-1.2.0...
    [ 1 of 23] Compiling Text.Regex.TDFA.NewDFA.Uncons ( Text/Regex/TDFA/NewDFA/Uncons.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Uncons.o )
    [ 2 of 23] Compiling Text.Regex.TDFA.IntArrTrieSet ( Text/Regex/TDFA/IntArrTrieSet.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/IntArrTrieSet.o )
    [ 3 of 23] Compiling Paths_regex_tdfa ( .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/autogen/Paths_regex_tdfa.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Paths_regex_tdfa.o )
    [ 4 of 23] Compiling Data.IntSet.EnumSet2 ( Data/IntSet/EnumSet2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntSet/EnumSet2.o )
    [ 5 of 23] Compiling Data.IntMap.EnumMap2 ( Data/IntMap/EnumMap2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntMap/EnumMap2.o )
    [ 6 of 23] Compiling Data.IntMap.CharMap2 ( Data/IntMap/CharMap2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntMap/CharMap2.o )
    [ 7 of 23] Compiling Text.Regex.TDFA.Common ( Text/Regex/TDFA/Common.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/Common.o )
    [ 8 of 23] Compiling Text.Regex.TDFA.Pattern ( Text/Regex/TDFA/Pattern.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/Pattern.o )
    [ 9 of 23] Compiling Text.Regex.TDFA.ReadRegex ( Text/Regex/TDFA/ReadRegex.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/ReadRegex.o )
    [10 of 23] Compiling Text.Regex.TDFA.CorePattern ( Text/Regex/TDFA/CorePattern.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/CorePattern.o )
    [11 of 23] Compiling Text.Regex.TDFA.NewDFA.MakeTest ( Text/Regex/TDFA/NewDFA/MakeTest.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/MakeTest.o )
    [12 of 23] Compiling Text.Regex.TDFA.NewDFA.Tester ( Text/Regex/TDFA/NewDFA/Tester.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Tester.o )
    [13 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_FA ( Text/Regex/TDFA/NewDFA/Engine_FA.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_FA.o )
    [14 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_NC ( Text/Regex/TDFA/NewDFA/Engine_NC.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_NC.o )
    [15 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_NC_FA ( Text/Regex/TDFA/NewDFA/Engine_NC_FA.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_NC_FA.o )
    [16 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine ( Text/Regex/TDFA/NewDFA/Engine.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine.o )


--  While building package texmath-0.8.2 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure (-9)
    Logs have been written to: /var/www/.stack-work/logs/texmath-0.8.2.log

    Configuring texmath-0.8.2...
    Building texmath-0.8.2...
    Preprocessing library texmath-0.8.2...
    [ 1 of 18] Compiling Text.TeXMath.Unicode.ToASCII ( src/Text/TeXMath/Unicode/ToASCII.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToASCII.o )
    [ 2 of 18] Compiling Text.TeXMath.TeX ( src/Text/TeXMath/TeX.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/TeX.o )
    [ 3 of 18] Compiling Text.TeXMath.Compat ( src/Text/TeXMath/Compat.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Compat.o )
    [ 4 of 18] Compiling Text.TeXMath.Readers.MathML.EntityMap ( src/Text/TeXMath/Readers/MathML/EntityMap.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/MathML/EntityMap.o )
    [ 5 of 18] Compiling Text.TeXMath.Readers.TeX.Macros ( src/Text/TeXMath/Readers/TeX/Macros.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/TeX/Macros.o )
    [ 6 of 18] Compiling Text.TeXMath.Types ( src/Text/TeXMath/Types.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Types.o )
    [ 7 of 18] Compiling Text.TeXMath.Shared ( src/Text/TeXMath/Shared.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Shared.o )
    [ 8 of 18] Compiling Text.TeXMath.Readers.MathML.MMLDict ( src/Text/TeXMath/Readers/MathML/MMLDict.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/MathML/MMLDict.o )
    [ 9 of 18] Compiling Text.TeXMath.Unicode.ToUnicode ( src/Text/TeXMath/Unicode/ToUnicode.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToUnicode.o )
    [10 of 18] Compiling Text.TeXMath.Unicode.ToTeX ( src/Text/TeXMath/Unicode/ToTeX.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToTeX.o )


--  While building package transformers-base-0.4.4 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure 1
    Logs have been written to: /var/www/.stack-work/logs/transformers-base-0.4.4.log

    Configuring transformers-base-0.4.4...
    Building transformers-base-0.4.4...
    Preprocessing library transformers-base-0.4.4...
    [1 of 1] Compiling Control.Monad.Base ( src/Control/Monad/Base.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Control/Monad/Base.o )
    .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/src/Control/Monad/Base.dump-hi: commitBuffer: invalid argument (invalid character)

It could happens at 3 or 4 / n process. When I restart stack build everythink is working then fail again after a few packages installed.

Do you have an idea?
Thanks

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 15, 2015

What does the locale command on your system output? And can you try running export LANG=en_US.UTF-8 before running stack?

@thelmstedt

This comment has been minimized.

thelmstedt commented Aug 17, 2015

I had this same issue and this advice fixed it for me. Here's what locale was set to:

LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Aug 17, 2015

I've just sent a GHC patch to address this:

https://phabricator.haskell.org/D1151

@borsboom In the short term, should we set the LANG env var in stack before calling Setup.hs build?

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 17, 2015

Given we're doing the equivalent in Windows, probably makes sense. It should probably append the encoding if there isn't already one, or replace it if there already is.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 17, 2015

Hmm, actually maybe always using C.UTF-8 for GHC output would be better (and easier).

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Aug 17, 2015

Perhaps what we should do is:

  • Have GHC provide output in UTF-8
  • When logging to a file, we get UTF-8 encoded files
  • When logging to the console, first decode that data to Text, and then use standard textual output to get the correct character encoding. We can use transliteration for unhandled characters, as I just did in https://phabricator.haskell.org/D1153
@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 21, 2015

Fix that sets LC_ALL for GHC pushed to master.

@borsboom borsboom closed this Aug 21, 2015

@MatthiasKauer

This comment has been minimized.

MatthiasKauer commented Nov 13, 2015

I just want to add to this and the discussion over on the mailing list: https://mail.haskell.org/pipermail/haskell-cafe/2015-October/121847.html

Setting

export LANG=en_US.UTF-8

was not sufficient on my Ubuntu 12.04 server for compiling gitit. It used lts
I also tried upgrading

resolver: lts-3.1

to lts-3.13 since you mentioned that you fixed things upstream but that didn't seem to help either. Using

export LC_ALL=C.UTF-8

worked however.

I also used stack clean a number of times in between although I have no idea if that improved things in any way.

The question is now: How would I upgrade to that new GHC version you mention cleanly? Or should I wait for the package managers of the individual repos? Or do I misunderstand and GHC version isn't tied to that resolver parameter?

Another question: Does stack help with static linking in any way? I really would've liked to build gitit on my notebook and then upload the binary, but I just couldn't make it work with cabal.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Nov 18, 2015

@MatthiasKauer

This comment has been minimized.

MatthiasKauer commented Nov 20, 2015

matthias@ugvps:~$ stack --version
Version 0.1.6.0, Git revision e22271f5ce9afa2cb5be3bad9cafa392c623f85c (2313 commits) x86_64

I'm running on a vserver, so locale may look odd.

matthias@ugvps:~/dvl/gitit$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_SG.utf8
LANGUAGE=
LC_CTYPE="en_SG.utf8"
LC_NUMERIC="en_SG.utf8"
LC_TIME="en_SG.utf8"
LC_COLLATE="en_SG.utf8"
LC_MONETARY="en_SG.utf8"
LC_MESSAGES="en_SG.utf8"
LC_PAPER="en_SG.utf8"
LC_NAME="en_SG.utf8"
LC_ADDRESS="en_SG.utf8"
LC_TELEPHONE="en_SG.utf8"
LC_MEASUREMENT="en_SG.utf8"
LC_IDENTIFICATION="en_SG.utf8"
LC_ALL=

Actually, this appears to be the same locale that I also have on my notebook (where the build succeeded, although not through stack but with cabal directly).
Is there a chance that locale is executed locally and not through ssh?

Will it help if I run further checks?

@borsboom

This comment has been minimized.

Contributor

borsboom commented Nov 22, 2015

@MatthiasKauer: Which Linux distribution are you using? And what are the values of the the LANG, LANGUAGE, and LC_ALL environment variables.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Nov 22, 2015

Just to add: I've tried this on Ubuntu, setting export LANG=en_SG.utf8 (which produced identical locale output to yours) and successfully been able to see error messages that would normally cause this type of error. If I don't actually have the en_SG.utf8 locale installed, the special characters are transliterated to ASCII, and after I did install the locale, the special characters were shown correctly.

Does you project's source code contain any Unicode characters? Would it be possible for you to share it to aid with reproduction?

Also, have you checked that the en_SG.utf8 locale is installed? Usually those first three messages in the locale output don't appear if the locale is there. You can see a list of installed locales using locale -a. On Ubuntu, you can install the locale using sudo apt-get install language-pack-en (not sure about other distros).

@MatthiasKauer

This comment has been minimized.

MatthiasKauer commented Nov 23, 2015

Thanks for looking into this!

This is a virtual server running Ubuntu 12.04. It's under OpenVZ and I'm not sure whether the hosting company would let me upgrade by now. Last time I asked they warned me about kernel issues....
Anyhow.

matthias@ugvps:~/dvl/gitit$ echo $LANG
en_SG.utf8
matthias@ugvps:~/dvl/gitit$ echo $LANGUAGE

matthias@ugvps:~/dvl/gitit$ echo $LC_ALL

I'm not quite sure why the locale is set that way there as I said. I'm in Singapore but I most probably wouldn't have bothered to tinker with that on a vserver.

Your other comment:

matthias@ugvps:~/dvl/gitit$ locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.UTF-8
POSIX

So I'm running a locale that I don't have installed, am I?
I still wonder whether the terminal sets that locale b/c my notebook has it.

The project in question is not mine. I was trying to compile https://github.com/jgm/gitit for deployment. I believe however, that the error came from a dependency. Here is the output snippet that I have stored.

    Configuring transformers-base-0.4.4...
    Building transformers-base-0.4.4...
    Preprocessing library transformers-base-0.4.4...
    [1 of 1] Compiling Control.Monad.Base ( src/Control/Monad/Base.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Control/Monad/Base.o )

    src/Control/Monad/Base.hs:22:1: Warning:
        The import of `Data.Monoid' is redundant
          except perhaps to import instances from `Data.Monoid'
        To import instances alone, use: import Data.Monoid()

    src/Control/Monad/Base.hs:24:1: Warning:
        The import of `Control.Applicative' is redundant
          except perhaps to import instances from `Control.Applicative'
        To import instances alone, use: import Control.Applicative()
    .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/src/Control/Monad/Base.dump-hi: commitBuffer: invalid argument (invalid character)
@borsboom

This comment has been minimized.

Contributor

borsboom commented Nov 24, 2015

Stack should work just fine on Ubuntu 12.04. Do you have root access to the machine? I wonder if running sudo locale-gen en_SG.utf8 makes a difference.

@MatthiasKauer

This comment has been minimized.

MatthiasKauer commented Nov 30, 2015

That seems to make a difference too.
I ran stack clean, got the same error building (although somewhere else, probably b/c I didn't clean dependencies too), entered sudo locale-gen en_SG.utf8 and built successfully.
Hope this helps.

@domenkozar

This comment has been minimized.

Contributor

domenkozar commented Sep 3, 2018

Since this is the first hit on google, I'll note how to fix this issue for Nix. Using custom shell.nix:

with (import <nixpkgs> {});

haskell.lib.buildStackProject {
  name = "myenv";

  buildInputs = [
    zlib
    ...
    # avoid on circleci: commitBuffer: invalid argument (invalid character)
    glibcLocales
  ];

  # avoid on circleci: commitBuffer: invalid argument (invalid character)
  LANG = "en_US.utf8";
}

As nix-shell: nix-shell -p glibcLocales --run "LANG=en_US.utf8 CMD"

@qrilka

This comment has been minimized.

Contributor

qrilka commented Sep 3, 2018

@domenkozar probably you have some good ideas how to add that LANG into https://github.com/commercialhaskell/stack/blob/master/src/Stack/Nix.hs#L83 - the problem applies also to #4095

@domenkozar

This comment has been minimized.

Contributor

domenkozar commented Sep 3, 2018

Hmm, since LANG=C is a default, we might as well just change that? Workaround is STACK_IN_NIX_EXTRA_ARGS: LANG=en_US.UTF-8.

@qrilka

This comment has been minimized.

Contributor

qrilka commented Sep 3, 2018

@domenkozar STACK_IN_NIX_EXTRA_ARGS is about a different thing (passing extra flags) but adding ,"LANG=en_US.UTF-8;" beside it looks to be a good enough workaround, I thought maybe there is some neater way to pass this locale but probably it should be ok so no extra nix file will be required for Unicode output. Thanks

@mihaimaruseac

This comment has been minimized.

Contributor

mihaimaruseac commented Sep 8, 2018

I think #4294 closes this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment