Lint error: hGetContents: invalid argument (invalid byte sequence) (Win10 + Atom)

[andregpss]

HIE is not working.

When i open an ".hs" file on Atom, the following error happens:

lint: C:\Users\Gostavo\AppData\Local\Temp\haskell-lsp15648\ComponentsMonitor.hs-00000-1623753167.hs: hGetContents: invalid argument (invalid byte sequence)

When i open the same project on VS Code, the error is:

The Haskell HIE (FirmMT) server crashed 5 times in the last

On HIE log, the last lines are:

2020-01-06 17:58:15.1511117 [ThreadId 3] - ---> {"jsonrpc":"2.0","id":6,"method":"textDocument/hover","params":{"textDocument":{"uri":"file:///C:/Users/Gostavo/Dropbox/MyP/src/Configuration/ComponentsMonitor.hs"},"position":{"line":0,"character":19}}}
2020-01-06 17:58:15.1511117 [ThreadId 31] - ****** reactor: got message number:13
2020-01-06 17:58:15.1511117 [ThreadId 31] - reactor:got HoverRequest:RequestMessage {_jsonrpc = "2.0", _id = IdInt 6, _method = TextDocumentHover, _params = TextDocumentPositionParams {_textDocument = TextDocumentIdentifier {_uri = Uri {getUri = "file:///C:/Users/Gostavo/Dropbox/MyP/src/Configuration/ComponentsMonitor.hs"}}, _position = Position {_line = 0, _character = 19}, _workDoneToken = Nothing}}
2020-01-06 17:58:15.1521097 [ThreadId 31] - reactor:HoverRequest done
2020-01-06 17:58:15.1521097 [ThreadId 37] - ideDispatcher: got request 13 with id: IdInt 6
2020-01-06 17:58:15.1521097 [ThreadId 31] - ****** reactor: top of loop
2020-01-06 17:58:15.1531533 [ThreadId 37] - newTypeCmd: (Position {_line = 0, _character = 19},Uri {getUri = "file:///C:/Users/Gostavo/Dropbox/MyP/src/Configuration/ComponentsMonitor.hs"})
2020-01-06 17:58:15.1571106 [ThreadId 37] - ideDispatcher: top of loop
2020-01-06 17:58:15.1571106 [ThreadId 4] - <--2--{"result":{"contents":{"kind":"markdown","value":""}},"jsonrpc":"2.0","id":6}
2020-01-06 17:58:18.4547533 [ThreadId 3] - ---> {"jsonrpc":"2.0","method":"textDocument/didClose","params":{"textDocument":{"uri":"file:///C:/Users/Gostavo/Dropbox/MyP/src/Configuration/ComponentsMonitor.hs"}}}
2020-01-06 17:58:18.4547533 [ThreadId 3] - Closed: Uri {getUri = "file:///C:/Users/Gostavo/Dropbox/MyP/src/Configuration/ComponentsMonitor.hs"}
2020-01-06 17:58:18.4557512 [ThreadId 3] - ---> {"jsonrpc":"2.0","id":7,"method":"shutdown","params":null}
2020-01-06 17:58:18.4557512 [ThreadId 31] - ****** reactor: got message number:14
2020-01-06 17:58:18.4557512 [ThreadId 31] - ****** reactor: processing NotDidCloseTextDocument
2020-01-06 17:58:18.4557512 [ThreadId 31] - ****** reactor: top of loop
2020-01-06 17:58:18.4557512 [ThreadId 4] - <--2--{"result":null,"jsonrpc":"2.0","id":7}
2020-01-06 17:58:18.4557512 [ThreadId 36] - ghcDispatcher:got request 14 with id: Nothing

I am using Atom 1.42.0 on Windows 10.

[fendor]

@andregpss All logs would be more interesting to see if some thread crashed, whether you were using stack or cabal, etc...
Does your file contain any unicode characters, maybe?

[andregpss]

@fendor I do not know whats happening, but when i openned Atom again, the error message changed.
The error now is:

"gcc.exe' failed in phase C pre-processor'. (Exit code: 1)"

Here it goes the full HIE log:
hie.log

[jneira]

When i open the same project on VS Code, the error is:

The Haskell HIE (FirmMT) server crashed 5 times in the last...

There is a bug in the vscode extension prior to 0.34 so i would recommend to update it.

[andregpss]

There is a bug in the vscode extension prior to 0.34 so i would recommend to update it.

@jneira I am already using 0.34 version of Haskell Language Server extension on VS Code.

The error messages on VS Code are now the same as those that appear in Atom:

• Got error while processing diagnostics: gcc.exe' failed in phase C pre-processor'. (Exit code: 1)

• lint: C:\Users\Gostavo\AppData\Local\Temp\haskell-lsp14376\ServerMain.hs-00000-643698865.hs: hGetContents: invalid argument (invalid byte sequence)

Does your file contain any unicode characters, maybe?

@fendor I do not think i am using unicode characters. That message pops up for every hs files i open.

[fendor]

Weird, can you share the project? The error message is unfortunately not very helpful.
EDIT: Maybe a subset of the project, if it is confedential? Or does any stack project yield this behaviour for you?

[andregpss]

EDIT: Maybe a subset of the project, if it is confedential? Or does any stack project yield this behaviour for you?

My project is relatively confidential.

I also installed HIE on Ubuntu Linux using Windows Subsystem, but the error persists.
I used the following tutorial: http://marco-lopes.com/articles/Vim-and-Haskell-in-2019/
Here it goes an screenshot (the error is on the bottom):

[fendor]

In the comment there is some non-ascii character é.
While hie should not crash on that, would you mind trying to remove that character, and any other such character in the comments?

[andregpss]

While hie should not crash on that, would you mind trying to remove that character, and any other such character in the comments?

I removed the é character but the error persists.

I openned now a very simple code file, but HIE still did not work.
Here is the screenshot:

[fendor]

Ok, I tried myself on windows 10, latest master, hie 8.6.5:

module Marshaller where 

import Data.Binary (decode, encode, Binary)
import Data.ByteString (ByteString)
import Data.ByteString.Lazy (fromStrict, toStrict)

marshall :: Binary a => a -> [ByteString]
marshall d = [(toStrict . encode) d]

unmarshall :: Binary a => [ByteString] -> a
unmarshall = decode . fromStrict . head

And everything seems to work, although there are some suspicous log messages.

Well, some semi-random debug ideas: would you mind trying to use cabal? You are using cabal if you add a cabal.project file and have cabal installed. Is the problem still the same?

Can you run hie in debug mode, e.g. hie --lsp --debug instead of only hie --lsp?

If your project is simple enough (only one exe and lib), you can also try the explicit project configuration, to do that, add a file named hie.yaml to your porject root with the contents:

cradle:
  stack:

Or, more complex:

cradle:
  stack:
    component: "project:lib"

Where project:lib is the library component of your project. You can find the available components with stack ide targets.

Last idea, not sure if it is important, did you maybe install hie over WSL and are using it without WSL?

[andregpss]

I openned an hs file external to my project and HIE worked! It really is an amazing tool! It was on neo-vim over WSL.
But the error continues with files from my project.
So, i think the problem is really with the configuration of my project.

Well, some semi-random debug ideas: would you mind trying to use cabal? You are using cabal if you add a cabal.project file and have cabal installed. Is the problem still the same?

I have a .cabal file, but not a cabal.project. Is it really necessary? How can i create it?

Can you run hie in debug mode, e.g. hie --lsp --debug instead of only hie --lsp?

How can i do that on linux with neo-vim?

add a file named hie.yaml to your porject root

Is it really necessary when using a .cabal file?

did you maybe install hie over WSL and are using it without WSL?

I have two scenarios on the same computer:

1. Over WSL: hie on neo-vim
2. Without WSL: hie on Atom

Here is the .cabal and the stack.yaml files i use on my project:
Proje.zip

[fendor]

I have a .cabal file, but not a cabal.project. Is it really necessary? How can i create it?

That or delete your local stack.yaml. Just something like echo "packages: ./" > cabal.project is enough, iirc.

Can you run hie in debug mode, e.g. hie --lsp --debug instead of only hie --lsp?

I dont know, sorry, not using neovim :(

Is it really necessary when using a .cabal file?

It is not, it is an experiment.

I have two scenarios on the same computer:

Ok, so probably not the culprit.

[fendor]

Are you maybe running into #1519?
Try to build your project by hand before opening it.

[andregpss]

HIE now works on my project, using Neo Vim on Ubuntu+WSL!
I noticed that Coc plugin configuration was different between the tutorial mentioned above and HIE README.md. I was using the tutorial version.
I noticed that on rootPatterns section, there was not *.cabal. When i put it, HIE worked! My packages are all declared on a Cabal file.
As also mentioned by @fendor , the project must have been build.
Almost everything works fine, except for the LanguageClient-neovim plugin, but i think this is another issue.
I am going to continue to try to make HIE work on Atom to answer the main issue.

[andregpss]

HIE is finally working on Atom + Windows 10!

Regarding the main issue of the current thread, the error message ´Lint error: hGetContents: invalid argument (invalid byte sequence)´ was caused by the use of UTF-8 characters on hs files.

Maybe if HIE pass --utf8 parameter to hlint will fix the problem, as stated on HLint issues thread.
By the way, is it possible for HIE users to pass that parameter to hlint?

However, i think that the following steps i've done were also necessary to make HIE works:

1. I was sharing the same project with Atom+Windows and NeoVim + Ubuntu + WSF. The build was being done on Ubuntu. So, Atom was using that build, instead of a Windows build.
2. As @fendor suggested, i created a hie.yaml that declares the two executables of the project. So, i also have on my projet a project.cabal and stack.yaml files.

The following HIE features still do not work on atom: apply-refact, Jump to definition, List all top level definitions.
Besides that, the LanguageClient plugin does not work on NeoVIM, as shown on the figure below.
I will try to solve those problems or maybe open another thread.

[fendor]

Thanks for finding this out! I didnt know we need to pass a flag to hlint to activate utf-8! We should probably pass that flag always?

It does not work without hie.yaml right?

[andregpss]

Thanks for finding this out! I didnt know we need to pass a flag to hlint to activate utf-8! We should probably pass that flag always?

I dont know. Due to Portuguese idiom, i have used some UTF-8 characters on code comments. I think it is a good idea to test the use of that flag because it can avoid that kind of error. However, it is not an error provoked by HIE, but by HLint. If we use HLint external to HIE, the error also occurs.

It does not work without hie.yaml right?

Right. HIE just dont work, but the invalide byte sequence error is not related to this.

[fendor]

Can you link logs for using hie without hie.yaml? In debug mode, if possible, e.g. hie --debug --lsp?

[andregpss]

Can you link logs for using hie without hie.yaml? In debug mode, if possible, e.g. hie --debug --lsp?

I tried to execute HIE on debug mode (hie --debug --lsp) using Atom HIE package settings (see image below), but an error occurs when starting Atom (see other image). In this scenario, the log is not even created.
Is this really how you run HIE on debug mode?

The log created on normal mode (no debug) is here:
hie-nodebug.log

[fendor]

I think you need a custom wrapper, e.g. hie-wrapper-debug

C:/bin/hie --debug --lsp

Or, I think you can also set the server trace to: verbose in the extensions.

[andregpss]

I am continuing the HIE test without hie.yaml, what makes to HIE not work.

I think you need a custom wrapper, e.g. hie-wrapper-debug

I dont know if i did it right. I created a hie-wrapper.bat file on Windows, and then configured the HIE extension on Atom by putting the full path on that file on custom hie-wrapper field.
Into hie-wrapper.bat, i wrote:

C:\Users\Gostavo\AppData\Roaming\local\bin\hie.exe --debug --lsp -l "c:\hie\hie-w.txt"

The HIE Wrapper generated the following log:
hie-w.txt

Atom also shown the error below:

[fendor]

Another idea: checkout the pr #1569, install this version of hie and execute it with hie --debug. This attempts to compile any file in the project (current directory) and it does not contain any confidential information like file contents, except for filepath information that you can easily strip out if you dont want to show that.
This may take a huge amount of time to execute, since it is not yet optimised, but it ought to contain any data necessary to debug this problem.
You can also execute the command with a modified environment variable CABAL_HELPER_DEBUG=4 to include even more information.
The whole output of that command is useful, so you can just copy-and-paste it here.

[andregpss]

install this version of hie

I did not undestand... What version are you talking about?

[fendor]

From the pr. Check it out, following the instructions on github and install it using the install script!

[fendor]

Exact instructions:

cd <hie-git-project>
git checkout -b fendor-extend-debug-utility master
git pull https://github.com/fendor/haskell-ide-engine.git extend-debug-utility
stack install.hs latest # or `./cabal-hie-install latest` or whatever ghc version you need

cd <project-to-build>
hie --debug

[andregpss]

Sorry... i was very busy past weeks and abandoned this discussion.

cd
git checkout -b fendor-extend-debug-utility master
git pull https://github.com/fendor/haskell-ide-engine.git extend-debug-utility
stack install.hs latest # or ./cabal-hie-install latest or whatever ghc version you need

is this test still worth it? if you want it, i can perform it now.

[fendor]

Yes, but you can use just the current master, no need for checking out the pr.

[andregpss]

Yes, but you can use just the current master, no need for checking out the pr.

Is the current master the default instalation (deacribed on readme.md)?

[jneira]

Maybe if HIE pass --utf8 parameter to hlint will fix the problem, as stated on HLint issues thread.
By the way, is it possible for HIE users to pass that parameter to hlint?

We are using hlint as a library but i hope we could tell to use utf8.

[andregpss]

I updated HIE today to check the question discussed above.
So, i deleted hie.yam from the root project and detected that, yes, HIE now works without that file.

I think HIE is working better on Windows 10 with the new version . The type information appears very fast now.

I am getting the same error opening cabal-helper

@jneira The UTF problem persists. On the other hand, HIE now shows the line where the error is, as the below figure shows:

[jneira]

@andregpss finally the fix has been done in the hie dep haskell-lsp (hlint assumes by default that file contents are in utf8)
I've tried a hie branch that uses the patched version of haskell-lsp and now cabal-helperworks for me.
If you have the chance of test that branch feel free to reopen if you continue experiencing the issue.

I dont have lexical errors with utf8 strings (but i did not have them before the patch that only fixes hlint) Just in case, are you using utf8 encoding for your files (you can check it in the state bar of vscode)?

[andregpss]

Just in case, are you using utf8 encoding for your files (you can check it in the state bar of vscode)?

Yes, i'm using utf-8 encoding (i saw this on vscode).

If you have the chance of test that branch feel free to reopen if you continue experiencing the issue.

How can i update HIE to use that branch? is it the last master commit? can i just pull the updates on HIE project?

[jneira]

@andregpss yeah (to the last question), you can checkout the branch in the hie project root dir:

> git checkout origin fix-origin-utf8
> stack install.hs hie-8.6.5 # or cabal-hie-install latest

[andregpss]

The UTF-8 problem persist, the same way as before.

git checkout origin fix-origin-utf8

This command did not work. I used git pull on hie directory. It showed the utf-8 fix update branch. Then, stack install.hs hie-8.6.5.

The HIE version i am using now is:

Version 1.0.0.0, Git revision "dc03b0b453dff503005804fe0bba04fc8b8d95f5" (3736 commits) x86_64 ghc-8.6.5

Also, i wrote before that HIE now works without hie.yaml. I have noticed that this is not true. When that file is not on the project root, the stack initialization message never stops. Then, when i put hie.yaml on project, and HIE worked again after restarting VSCode.

[jneira]

oh sorry for my incomplete and erroneous steps:

D:\ws\haskell\haskell-ide-engine>git status
On branch master
Your branch is up to date with 'upstream/master'.
nothing to commit, working tree clean

D:\ws\haskell\haskell-ide-engine>git log -n 1
commit dc03b0b453dff503005804fe0bba04fc8b8d95f5 (HEAD -> master, origin/master
, upstream/HEAD)
...............

D:\ws\haskell\haskell-ide-engine>git remote -v | grep haskell/haskell-ide-engine

origin        https://github.com/haskell/haskell-ide-engine (fetch)
origin        https://github.com/haskell/haskell-ide-engine (push)

D:\ws\haskell\haskell-ide-engine>git checkout -t origin/fix-hlint-utf8
Switched to a new branch 'fix-hlint-utf8'
Branch 'fix-hlint-utf8' set up to track remote branch 'fix-hlint-utf8' from 'origin'.

D:\ws\haskell\haskell-ide-engine>git log -n 1
commit b1f50edc095d454087e2e34e9419922e2d248f15 (HEAD -> fix-hlint-utf8, origin/fix-hlint-utf8)
Author: jneira <atreyu.bbb@gmail.com>
Date:   Wed Jan 29 10:04:07 2020 +0100

    Use haskell-lsp with utf8 fix

D:\ws\haskell\haskell-ide-engine>stack install.hs hie-8.6.5

Also, i wrote before that HIE now works without hie.yaml. I have noticed that this is not true. When that file is not on the project root, the stack initialization message never stops. Then, when i put hie.yaml on project, and HIE worked again after restarting VSCode.

Sorry to hear that, could you attach the log with verbose mode on using the fix-hlint-utf8 version?

[andregpss]

@jneira The command git checkout -t origin/fix-hlint-utf8 still not work.
Here's my commands:

c:\hie\haskell-ide-engine>git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

c:\hie\haskell-ide-engine>git log -n 1
commit dc03b0b (HEAD -> master, origin/master, origin/HEAD)
Merge: 62f38f8 40483ba
Author: Javier Neira atreyu.bbb@gmail.com
Date: Wed Jan 29 22:56:07 2020 +0100

Merge pull request #1615 from jneira/stack-install-dev

Add dev target to stack install.hs

c:\hie\haskell-ide-engine>git remote -v | grep haskell/haskell-ide-engine
origin https://github.com/haskell/haskell-ide-engine (fetch)
origin https://github.com/haskell/haskell-ide-engine (push)
c:\hie\haskell-ide-engine>git checkout -t upstream/fix-hlint-utf8
fatal: 'upstream/fix-hlint-utf8' is not a commit and a branch 'fix-hlint-utf8' cannot be created from it

c:\hie\haskell-ide-engine>git --version
git version 2.24.0.windows.2

[jneira]

In the log i see git checkout -t upstream/fix-hlint-utf8 but it should be origin/fix-hlint-utf8 (like in the start of your comment)

[alanz]

I am planning on getting an updated haskell-lsp release out soon.

…

On Thu, 30 Jan 2020 at 18:04, Javier Neira ***@***.***> wrote: In the log i see git checkout -t upstream/fix-hlint-utf8 but it should be origin/fix-hlint-utf8 (like in the start of your comment) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1560?email_source=notifications&email_token=AADEAB3BQLLHNU4VABBEOW3RAMJETA5CNFSM4KDK6K3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKL6NUQ#issuecomment-580380370>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADEABYAQ7VSHKDFAWE62YLRAMJETANCNFSM4KDK6K3A> .

[andregpss]

In the log i see git checkout -t upstream/fix-hlint-utf8 but it should be origin/fix-hlint-utf8 (like in the start of your comment)

@jneira I'm sorry. Now using fix-hlint-utf8 and UTF-8 issue is solved! Thanks.
Is it advisable to come back to master branch now? how can i do it?

Sorry to hear that, could you attach the log with verbose mode on using the fix-hlint-utf8 version?

@jneira I deleted hie.yaml and restarded VSCode on Windows 10. Here's the log after restarting:

log-WithoutHie.txt

[jneira]

@andregpss great! dont worry about, those thing happens continually

I'll reopen this and close it when hie start to use the lsp-haskell version with the fix

[jneira]

@andregpss current master already has the lsp-haskell version with the fix so

> git checkout master
> git pull
> stack install.hs hie-8.6.5

should install a hie version that works
Feel free to reopen if you continue experiencing the issue.

[andregpss]

I updated HIE to the master version, following the steps above indicated by @jneira, and the UTF-8 issue happened again.

Here is my HIE version:

Version 1.0.0.0, Git revision caf7223 (3765 commits) x86_64 ghc-8.6.5

[jneira]

@andregpss wow, however, afaiu the hlint one is gone definitely, right?
The utf-8 issue is a different one (although related) and i will open a new issue, if you dont mind.

[andregpss]

@jneira the current error message is the same as the title of this issue:

lint: C:\Users\Gostavo\AppData\Local\Temp\haskell-lsp21340\ClientMain.hs-00000--33151423.hs: hGetContents: invalid argument (invalid byte sequence)

The difference between when i opend this thread and now is that there is a second error message, that i sent on a figure attached in a message above:

lexical error in string/character literal (UTF-8 decoding error)

I warn that when i used fix-hlint-utf8 branch, the error ended. When i changed to master branch, the error returned.
Has the master branch really been updated?

[fendor]

@andregpss Assuming you pulled in the latest changes, yeah, it should be.

[jneira]

errr, it seems the fix is in haskell-lsp-0.20 and master still uses 0.19 😞

haskell-ide-engine/haskell-ide-engine.cabal

Line 76 in 7283ae2

, haskell-lsp == 0.19.*

@andregpss sorry for the inconveniences, i thought that we already have bumped the version

[andregpss]

@jneira No problem! Please warn me when you want me to test a new HIE version.

[jneira]

Hi again @andregpss, i think this time will be the right one. Pull again from master and it should work.

[andregpss]

@jneira UTF-8 issue fixed on master branch! thanks a lot!