Make fails with UnicodeDecodeError #830

Spielzeug · 2017-09-24T12:49:24Z

Under archlinux, make fails either using python2 or python3:

cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DPYTHON_EXECUTABLE=/usr/bin/python2 -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_LIBRARY=/usr/lib/libpython2.7.so -DPYTHON_BASENAME=-python2.7 ../nest-2.12.0

OR

cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DPYTHON_EXECUTABLE=/usr/bin/python -DPYTHON_INCLUDE_DIR=/usr/include/python3.6 -DPYTHON_LIBRARY=/usr/lib/libpython3.6.so -DPYTHON_BASENAME=-python3.6 ../nest-2.12.0

CMake output: https://pastebin.com/3esMhsM7

after this make command fails with:

Traceback (most recent call last):
File "generate_help.py", line 76, in
filetext = f.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1128: ordinal not in range(128)
make[2]: *** [doc/CMakeFiles/generate_help.dir/build.make:58: generate_help] Error 1
make[1]: *** [CMakeFiles/Makefile2:216: doc/CMakeFiles/generate_help.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

Tested the generate_help.py invokation with python2 and python3, it works with python2. Had to change cmake generated build.make:
generate_help: doc/CMakeFiles/generate_help.dir/build.make
@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold "Extracting help information; this may take a litte while."
cd /home/rei/Downloads/nest-2.12.0/extras/help_generator && python -B generate_help.py /home/rei/Downloads/nest-2.12.0 /home/rei/Downloads/nest-build
cd /home/rei/Downloads/nest-2.12.0/extras/help_generator && python -B generate_helpindex.py /home/rei/Downloads/nest-build/doc
.PHONY : generate_help

to

generate_help: doc/CMakeFiles/generate_help.dir/build.make
@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold "Extracting help information; this may take a litte while."
cd /home/rei/Downloads/nest-2.12.0/extras/help_generator && python2 -B generate_help.py /home/rei/Downloads/nest-2.12.0 /home/rei/Downloads/nest-build
cd /home/rei/Downloads/nest-2.12.0/extras/help_generator && python2 -B generate_helpindex.py /home/rei/Downloads/nest-build/doc
.PHONY : generate_help

Can someone point out what should I have done to avoid this messy change?

sanjayankur31 · 2017-09-27T23:39:08Z

It seems to be an encoding issue - explained quite well here. Funnily enough, I don't see it neither with python2 nor python3 on Fedora 26 here, so could be something to do with the system defaults there? Maybe check your default encodings? In a python interpreter, import sys and check the output of sys.getdefaultencoding()? I get ascii for python2 and utf-8 for python3 here.

Spielzeug · 2017-09-28T19:38:06Z

It is an encoding problem, my problem is that I do not know how to avoid it properly. I too have ascii for python2 and utf-8 for python3. For now I circumvented the problem as I said above and have nest installed and running, for the future it would be nice to know how to force python encoding during cmake configuration generation.

heplesser · 2017-10-03T21:51:38Z

@steffengraber Could you take a look at this? I assume the problem occurs because some .sli file contains non-ASCII characters. Then, if the file is opened using f = open(file, 'r'), this will lead to trouble in Python 3. We probably have to check two things here:

How to handle file opening properly under Py2 and Py3 (in Py3, add encoding='utf-8' as argument to open(), but that won't run under Py2.
Ensure that all NEST code files are UTF-8 encoded.

steffengraber · 2017-10-09T06:43:42Z

@heplesser We can use the io module and write io.open(“file”, “r”, encoding=”utf-8”) instead of open( ... ). It is available since 2.6 and standard in python3.
I have to test it.

heplesser · 2017-10-12T12:58:01Z

We can use the following to scan the code base for files that are not valid UTF-8 files:

for r, d, f in os.walk('.'):
       for fn in f:
            try:
                open(os.path.join(r, fn), 'r', encoding='utf-8').read()
            except:
                print('Error:', os.path.join(r, fn))

jougs · 2017-10-12T16:12:52Z

@heplesser: Thanks for the code. Here's my extended version, which should work with Python 2 and 3:

import os
from io import open

exclude_paths = [".git"]
exclude_files = [".jpg", ".pdf", ".png"]

for r, d, f in os.walk('.'):
    for fn in f:
       check = True
       for ep in exclude_paths:
           if r.startswith(os.path.join(".", ep)):
                check = False
                break
       for ef in exclude_files:
           if fn.endswith(ef):
                check = False
                break
       if check:
            try:
                open(os.path.join(r, fn), 'r', encoding='utf-8').read()
            except:
                print('Error:', os.path.join(r, fn))

On my machine (Ubuntu 17.04), this does not report any errors. @Spielzeug, can you please save the code to a file in the NEST source directory on your machine and execute it? Thanks!

planetmija · 2017-11-16T18:04:29Z

@jougs : I tried it with my own fresh compiled python 3.6.3, but I don't get any output:

$ python bla.py 
3.6.3 (default, Nov 16 2017, 17:53:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

OR

$ python bla.py 
3.5.3 |Intel Corporation| (default, Apr 27 2017, 18:08:47) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

But I get the same compile error:

File "/opt/bwhpc/common/devel/python/3.6.3.20171114/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]

EDIT: bla.py is your code + print(sys.version)

planetmija · 2017-11-17T13:01:48Z

@jougs : Sorry, forgot, that I already deleted the build dir. Here are the files:

3.6.3 (default, Nov 16 2017, 17:53:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
Error: ./CMakeFiles/feature_tests.bin
Error: ./CMakeFiles/CheckTypeSize/UINT16_T_SIZE.bin
Error: ./CMakeFiles/CheckTypeSize/LONG_LONG_SIZE.bin
Error: ./CMakeFiles/3.9.0/CMakeDetermineCompilerABI_CXX.bin
Error: ./CMakeFiles/3.9.0/CMakeDetermineCompilerABI_C.bin
Error: ./CMakeFiles/3.9.0/CompilerIdCXX/a.out
Error: ./CMakeFiles/3.9.0/CompilerIdC/a.out
Error: ./CMakeFiles/FindOpenMP/ompver_CXX.bin
Error: ./CMakeFiles/FindOpenMP/ompver_C.bin
Error: ./libnestutil/libnestutil.so
Error: ./libnestutil/CMakeFiles/nestutil.dir/stopwatch.cpp.o
Error: ./libnestutil/CMakeFiles/nestutil.dir/logging_event.cpp.o
Error: ./libnestutil/CMakeFiles/nestutil.dir/propagator_stability.cpp.o
Error: ./libnestutil/CMakeFiles/nestutil.dir/numerics.cpp.o

krisdamato · 2017-12-20T12:45:52Z

Any updates on this? I'm getting the same error when running make install on both Python 3.5.4 and 2.7 in Ubuntu 16.04, so the above workaround does not work.

steffengraber · 2018-01-09T07:00:12Z

@krisdamato
I tried it with a fresh install of Ubuntu 16.04.3 with all the latest updates (Python 2.7.12, Python 3.5.2) in a Virtualbox and could not reproduce your problems.
Additionally I tried Python 3.6.4. Even with that I could not create the error.

That's what I did in detail:

Install the dependencies:

sudo apt-get install -y build-essential cmake libltdl-dev libreadline6-dev libncurses5-dev libgsl-dev python-all-dev python-numpy python-scipy python-matplotlib ipython openmpi-bin libopenmpi-dev python-nose cython

Getting NEST

wget https://github.com/nest/nest-simulator/archive/master.zip
unzip master.zip
mkdir build
mkdir install
cd build

Installing NEST with Python 2

cmake -DCMAKE_INSTALL_PREFIX:PATH=../install -Dwith-python=2 -DPYTHON_EXECUTABLE=/usr/bin/python2 ../nest-simulator-master
make
make install

or: Installing NEST with Python 3

cmake -DCMAKE_INSTALL_PREFIX:PATH=../install -Dwith-python=3 -DPYTHON_EXECUTABLE=/usr/bin/python3 ../nest-simulator-master
make
make install

Could you please explain exactly what you did?

krisdamato · 2018-01-09T10:20:25Z

I was using an ubuntu image from their official Docker repository, creating a conda environment, and building. I can't recall the exact steps, because now I've fixed it by using a different image. My apologies. On an unrelated note: providing an official NEST simulator docker image would be really cool for people who need to set this up on private clusters/home machines.

…

On Tue, Jan 9, 2018 at 8:00 AM, graber ***@***.***> wrote: @krisdamato <https://github.com/krisdamato> I tried it with a fresh install of Ubuntu 16.04.3 with all the latest updates (Python 2.7.12, Python 3.5.2) in a Virtualbox and could not reproduce your problems. Additionally I tried Python 3.6.4. Even with that I could not create the error. That's what I did in detail: Install the dependencies: sudo apt-get install -y build-essential cmake libltdl-dev libreadline6-dev libncurses5-dev libgsl-dev python-all-dev python-numpy python-scipy python-matplotlib ipython openmpi-bin libopenmpi-dev python-nose cython Getting NEST wget https://github.com/nest/nest-simulator/archive/master.zip unzip master.zip mkdir build mkdir install cd build Installing NEST with Python 2 cmake -DCMAKE_INSTALL_PREFIX:PATH=../install -Dwith-python=2 -DPYTHON_EXECUTABLE=/usr/bin/python2 ../nest-simulator-master make make install or: Installing NEST with Python 3 cmake -DCMAKE_INSTALL_PREFIX:PATH=../install -Dwith-python=3 -DPYTHON_EXECUTABLE=/usr/bin/python3 ../nest-simulator-master make make install Could you please explain exactly what you did? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#830 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGjleHuN4vQrM3tgYXfaLosfwCK1qxgPks5tIw5-gaJpZM4Ph4Ml> .

steffengraber · 2018-01-11T07:18:21Z

@krisdamato An official docker image is in preparation.

steffengraber · 2018-01-11T07:19:46Z

I tried the installation with non-UTF-8 locales settings (en_US) of the system and could now recover the error message.
When I reset everything to en_US. UTF-8 make runs without error messages.

jougs · 2018-02-07T16:16:22Z

@steffengraber: can you please run the code from my comment with the non-unicode locale and see if there are files with errors? Thanks!

steffengraber · 2018-02-08T07:59:33Z

@jougs Sorry, checked again in new master with python 2 and 3. No errors.

terhorstd · 2018-03-05T11:09:27Z

@steffengraber : could you add an , encoding="utf8" to the open() statements in the help generation? This would probably fix this issue (and would be more correct procedure).

steffengraber · 2018-03-20T10:22:47Z

Because encoding='utf-8' as argument to open() won't run under Python2 I decided to use the io-module (io.open(file”, “r”, encoding=”utf-8”)).

heplesser · 2018-03-26T08:18:26Z

Fixed by #915.

heplesser assigned steffengraber Oct 3, 2017

steffengraber mentioned this issue Mar 20, 2018

Fix issue 830 - Make fails with UnicodeDecodeError #915

Merged

heplesser added ZP: PR Created DO NOT USE THIS LABEL and removed ZP: Pending DO NOT USE THIS LABEL labels Mar 20, 2018

heplesser closed this as completed Mar 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make fails with UnicodeDecodeError #830

Make fails with UnicodeDecodeError #830

Spielzeug commented Sep 24, 2017 •

edited

Loading

sanjayankur31 commented Sep 27, 2017

Spielzeug commented Sep 28, 2017

heplesser commented Oct 3, 2017

steffengraber commented Oct 9, 2017

heplesser commented Oct 12, 2017

jougs commented Oct 12, 2017 •

edited

Loading

planetmija commented Nov 16, 2017 •

edited

Loading

planetmija commented Nov 17, 2017

krisdamato commented Dec 20, 2017

steffengraber commented Jan 9, 2018

krisdamato commented Jan 9, 2018 via email

steffengraber commented Jan 11, 2018

steffengraber commented Jan 11, 2018

jougs commented Feb 7, 2018

steffengraber commented Feb 8, 2018

terhorstd commented Mar 5, 2018

steffengraber commented Mar 20, 2018

heplesser commented Mar 26, 2018

Make fails with UnicodeDecodeError #830

Make fails with UnicodeDecodeError #830

Comments

Spielzeug commented Sep 24, 2017 • edited Loading

sanjayankur31 commented Sep 27, 2017

Spielzeug commented Sep 28, 2017

heplesser commented Oct 3, 2017

steffengraber commented Oct 9, 2017

heplesser commented Oct 12, 2017

jougs commented Oct 12, 2017 • edited Loading

planetmija commented Nov 16, 2017 • edited Loading

planetmija commented Nov 17, 2017

krisdamato commented Dec 20, 2017

steffengraber commented Jan 9, 2018

krisdamato commented Jan 9, 2018 via email

steffengraber commented Jan 11, 2018

steffengraber commented Jan 11, 2018

jougs commented Feb 7, 2018

steffengraber commented Feb 8, 2018

terhorstd commented Mar 5, 2018

steffengraber commented Mar 20, 2018

heplesser commented Mar 26, 2018

Spielzeug commented Sep 24, 2017 •

edited

Loading

jougs commented Oct 12, 2017 •

edited

Loading

planetmija commented Nov 16, 2017 •

edited

Loading