codec can't decode byte 0xb3 in position 193: ordinal not in range(128)

[26O0]

Make sure to follow and check these boxes before submitting an issue! Thank you.

• Supply iocage --version
• Supply the commands used, along with any steps to recreate it.
• Provide the output from the command you issued.
• Supply what you expected the result or output to be
• Checked that the problem has not already been fixed on master if using
  a stable release.

Version 0.9.8.1 05/23/2017

ioage list

% iocage list
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 1063, in invoke
    Command.invoke(self, ctx)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click-6.7-py3.6.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage/main.py", line 102, in cli
    ioc_check.IOCCheck()
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py", line 19, in __init__
    self.__check_datasets__()
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py", line 33, in __check_datasets__
    list(pool.root.datasets)))) > 0
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py", line 32, in <lambda>
    has_duplicates = len(list(filter(lambda x: x.mountpoint == "/iocage",
  File "libzfs.pyx", line 2280, in libzfs.ZFSDataset.mountpoint.__get__ (libzfs.c:50748)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb3 in position 193: ordinal not in range(128)

Expected results:
I expect to have a list of jails returned

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[26O0]

Your template doesn't ask for how iocage is installed. This is how it was installed.

1. git clone https://github.com/trueos/freebsd /usr/src
2. pkg install python36 libgit2
3. git clone --recursive https://github.com/iocage/iocage
4. make install as root
5. To install subsequent updates: run make install as root.

[26O0]

Here's more:

% cat .login_conf 
# $FreeBSD: src/share/skel/dot.login_conf,v 1.3 2001/06/10 17:08:53 ache Exp $
#
# see login.conf(5)
#
me:\
        :charset=UTF-8:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:

% locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE=C
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

[skarekrow]

Can you post a zfs list as well? I have a feeling there's a special character or something. Since I don't recall decoding UTF8 anywhere in this code.

[26O0]

Can you post a zfs list as well? I have a feeling there's a special character or something. Since I don't recall decoding UTF8 anywhere in this code.

sure! ``` NAME USED AVAIL REFER MOUNTPOINT tank 18.7G 873G 88K none tank/ROOT 14.0G 873G 88K none tank/ROOT/12.0-CURRENT-up-20170420_212750 320K 873G 4.79G / tank/ROOT/12.0-CURRENT-up-20170421_211808 480K 873G 5.93G / tank/ROOT/12.0-CURRENT-up-20170425_235231 14.0G 873G 6.21G / tank/ROOT/initial 288K 873G 4.48G /mnt tank/iocage 476M 873G 96K /iocage tank/iocage/download 112M 873G 88K /iocage/download tank/iocage/download/11.0-RELEASE 111M 873G 111M /iocage/download/11.0-RELEASE tank/iocage/images 88K 873G 88K /iocage/images tank/iocage/jails 7.93M 873G 88K /iocage/jails tank/iocage/jails/5b8a1d6e-cd3b-4fe5-bb61-108080c08629 7.84M 873G 92K /iocage/jails/5b8a1d6e-cd3b-4fe5-bb61-108080c08629 tank/iocage/jails/5b8a1d6e-cd3b-4fe5-bb61-108080c08629/root 7.75M 873G 356M /iocage/jails/5b8a1d6e-cd3b-4fe5-bb61-108080c08629/root tank/iocage/log 92K 873G 92K /iocage/log tank/iocage/releases 356M 873G 88K /iocage/releases tank/iocage/releases/11.0-RELEASE 356M 873G 88K /iocage/releases/11.0-RELEASE tank/iocage/releases/11.0-RELEASE/root 356M 873G 355M /iocage/releases/11.0-RELEASE/root tank/iocage/templates 88K 873G 88K /iocage/templates tank/tmp 796K 873G 796K /tmp tank/usr 4.26G 873G 88K none tank/usr/home 1.89G 873G 88K /usr/home tank/usr/home/sean 1.89G 873G 1.89G /usr/home/sean tank/usr/jails 88K 873G 88K /usr/jails tank/usr/obj 88K 873G 88K /usr/obj tank/usr/ports 88K 873G 88K /usr/ports tank/usr/src 2.37G 873G 2.37G /usr/src tank/var 25.3M 873G 88K none tank/var/audit 88K 873G 88K /var/audit tank/var/log 24.9M 873G 24.9M /var/log tank/var/mail 112K 873G 112K /var/mail tank/var/tmp 136K 873G 136K /var/tmp ```

[skarekrow]

Damn, I don't see anything that's sticking out. I can't replicate it currently, but I'll keep trying. So you were able to create a jail though? Because this same routine would have been called, odd it didn't die for you then as well.

[26O0]

Hi @skarekrow,

I was previously using an iocage (also from git src), so a jail already created and that's why I want to list it now. I'll see if I can create a new jail with the latest iocage.

[skarekrow]

Creating a new jail should fail the same way as list then. Do you recall the last version you used?

[26O0]

Yes, failing in idential way.

I don't reall version but the /usr/local/bin/ioage date was from either the 8th or 18th of May. Probably the earlier date, but I don't know for certain.

[skarekrow]

That would be before the file was converted to libzfs if the 8th was the right date, so I'll summon @gronke and see if he has any ideas as well.

[26O0]

okay, I just updated trueOS base and packages and now getting a SO error.

% iocage list
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 6, in <module>
    from iocage.main import cli
  File "/usr/local/lib/python3.6/site-packages/iocage/main.py", line 13, in <module>
    import iocage.lib.ioc_check as ioc_check
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py", line 6, in <module>
    import iocage.lib.ioc_common
  File "/usr/local/lib/python3.6/site-packages/iocage/lib/ioc_common.py", line 12, in <module>
    import pygit2
  File "/usr/local/lib/python3.6/site-packages/pygit2/__init__.py", line 32, in <module>
    from _pygit2 import *
ImportError: Shared object "libgit2.so.24" not found, required by "_pygit2.so"

[skarekrow]

Just for fun @26O0, can you try editing /usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py and changing the line has_duplicates = len(list(filter(lambda x: x.mountpoint == "/iocage", to has_duplicates = len(list(filter(lambda x: x.mountpoint == u"/iocage",? Make sure to remove the pyc file if it exists, rm /usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.pyc

The SO error is because you need to rebuild pygit2, since you upgraded libgit2 to 0.25.1 I'd imagine.

[26O0]

The SO error is because you need to rebuild pygit2, since you upgraded libgit2 to 0.25.1 I'd imagine.

Yes, I now have libgit2-0.25.1_1

I cd into py-libzfs ran `python3.6 setup.py install, which installed but generated errors.

I went back to iocage dir and installed without issues, but running ioage still generating errors.

Do you recommend I still edit the files you listed above?

[skarekrow]

You're still getting the SO errors? You could try just upgrading pygit2 as well with pip install -U pygit2.

Yes, please try editing that file and add the u before /iocage, I'm interested to see what it says. Please post the stack trace if it occurs too 👍

[26O0]

Actually, if I make in ioage dir, I get this:

make
echo -e "import os\ntry:\n  if not os.listdir('/usr/src'): exit('/usr/src must be populated!')\nexcept FileNotFoundError:\n  exit('/usr/src must be populated!')" | python3.6
git pull
Already up-to-date.
python3.6 -m ensurepip
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages
Requirement already satisfied: pip in /usr/local/lib/python3.6/site-packages
pip3.6 install -U Cython
Requirement already up-to-date: Cython in /usr/local/lib/python3.6/site-packages
cd py-libzfs && python3.6 setup.py build && python3.6 setup.py install
running build
running build_py
running build_ext
skipping 'libzfs.c' Cython extension (up-to-date)
building 'libzfs' extension
cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fstack-protector -fno-strict-aliasing -fPIC -I/usr/src/cddl/lib/libumem -I/usr/src/sys/cddl/compat/opensolaris/ -I/usr/src/sys/cddl/compat/opensolaris -I/usr/src/cddl/compat/opensolaris/include -I/usr/src/cddl/compat/opensolaris/lib/libumem -I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common -I/usr/src/sys/cddl/contrib/opensolaris/common/zfs -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/sys -I/usr/src/cddl/contrib/opensolaris/head -I/usr/src/sys/cddl/contrib/opensolaris/uts/common -I/usr/src/cddl/contrib/opensolaris/lib/libnvpair -I/usr/src/cddl/contrib/opensolaris/lib/libuutil/common -I/usr/src/cddl/contrib/opensolaris/lib/libzfs/common -I/usr/src/cddl/contrib/opensolaris/lib/libzfs_core/common -I/usr/local/include/python3.6m -c libzfs.c -o build/temp.freebsd-12.0-CURRENT-amd64-3.6/libzfs.o -DNEED_SOLARIS_BOOLEAN -D_XPG6 -g -O0
libzfs.c:14747:13: warning: code will never be executed [-Wunreachable-code]
  __pyx_r = 0;
            ^
libzfs.c:19649:73: warning: incompatible pointer to integer conversion passing 'PyObject *' (aka 'struct _object *') to parameter of type 'zfs_type_t' [-Wint-conversion]
  __pyx_v_handle = zfs_path_to_zhandle(__pyx_v_self->handle, __pyx_t_1, __pyx_t_2);
                                                                        ^~~~~~~~~
/usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h:704:79: note: passing argument to parameter here
extern zfs_handle_t *zfs_path_to_zhandle(libzfs_handle_t *, char *, zfs_type_t);
                                                                              ^
libzfs.c:37575:13: warning: code will never be executed [-Wunreachable-code]
  __pyx_r = 0;
            ^
libzfs.c:47906:89: error: too few arguments to function call, expected 4, have 3
        __pyx_v_ret = lzc_send_space(__pyx_v_c_name, __pyx_v_cfromname, (&__pyx_v_space));
                      ~~~~~~~~~~~~~~                                                    ^
/usr/src/cddl/contrib/opensolaris/lib/libzfs_core/common/libzfs_core.h:72:1: note: 'lzc_send_space' declared here
int lzc_send_space(const char *, const char *, enum lzc_send_flags, uint64_t *);
^
libzfs.c:50839:74: warning: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_ret = zfs_destroy_snaps(__pyx_v_self->__pyx_base.handle, __pyx_v_c_name, 1);
                                                                         ^~~~~~~~~~~~~~
/usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h:564:52: note: passing argument to parameter here
extern int zfs_destroy_snaps(zfs_handle_t *, char *, boolean_t);
                                                   ^
4 warnings and 1 error generated.
error: command 'cc' failed with exit status 1
*** Error code 1

Stop.
make: stopped in /usr/home/sean/bin/iocage

If I pip3.6 install -U ., it installs.

[skarekrow]

Interesting, but since you just updated TrueOS, you may have an old and incompatible source tree now unless you also updated that.

[26O0]

gahhh! what's wrong with my system?!

% sudo pip3.6 install -U pygit2
The directory '/usr/home/sean/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/usr/home/sean/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pygit2
  Downloading pygit2-0.25.1.tar.gz (456kB)
    100% |████████████████████████████████| 460kB 1.5MB/s 
    Complete output from command python setup.py egg_info:
    
    Installed /tmp/pip-build-71z_sv35/pygit2/.eggs/cffi-1.9.1-py3.6-freebsd-12.0-CURRENT-amd64.egg
    /usr/local/lib/python3.6/site-packages/setuptools/extern/__init__.py:1: UserWarning: Module cffi was already imported from /usr/local/lib/python3.6/site-packages/cffi/__init__.py, but /tmp/easy_install-1_u9fk77/cffi-1.9.1 is being added to sys.path
      from pkg_resources.extern import VendorImporter
    /usr/local/lib/python3.6/site-packages/setuptools/dist.py:364: UserWarning: Module cffi was already imported from /usr/local/lib/python3.6/site-packages/cffi/__init__.py, but /tmp/pip-build-71z_sv35/pygit2/.eggs/cffi-1.9.1-py3.6-freebsd-12.0-CURRENT-amd64.egg is being added to sys.path
      pkg_resources.working_set.add(dist, replace=True)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-71z_sv35/pygit2/setup.py", line 210, in <module>
        **extra_args)
      File "/usr/local/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 318, in __init__
        _Distribution.__init__(self, attrs)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/usr/local/lib/python3.6/site-packages/setuptools/dist.py", line 376, in finalize_options
        ep.load()(self, ep.name, value)
      File "/usr/local/lib/python3.6/site-packages/cffi/setuptools_ext.py", line 188, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/usr/local/lib/python3.6/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/usr/local/lib/python3.6/site-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "pygit2/_run.py", line 67, in <module>
        ffi = FFI()
      File "/usr/local/lib/python3.6/site-packages/cffi/api.py", line 54, in __init__
        backend.__version__, backend.__file__))
    Exception: Version mismatch: this is the 'cffi' package version 1.10.0, located in '/usr/local/lib/python3.6/site-packages/cffi/api.py'.  When we import the top-level '_cffi_backend' extension module, we get version 1.9.1, located in '/tmp/pip-build-71z_sv35/pygit2/.eggs/cffi-1.9.1-py3.6-freebsd-12.0-CURRENT-amd64.egg/_cffi_backend.so'.  The two versions should be equal; check your installation.
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-71z_sv35/pygit2/

[skarekrow]

There's nothing wrong ;), I had a brain fart, you can safely remove cffi as well. pygit2 will bring that back in.

So to recap:

pip3.6 uninstall cffi ; pip3.6 install -U pygit2`

[26O0]

unless you also updated that.

yeah, unfortunately that's not it. I checked out the src and there hasn't been any updates since.

[26O0]

I had a brain fart

that's understandable, it's the middle of the night!

Now I'm getting the original error and will edit the files you recommended.

[skarekrow]

Hmm @kmoore134 may not have updated the tree or something. I typically see that error when I have an old source tree. You can try opening up a libzfs ticket on @jceel's tracker, he's very responsive and will likely get to the bottom of it faster then myself.

He may even have an idea of this UTF-8 issue, as it may stem from libzfs instead of iocage as well.

[skarekrow]

@26O0 looks like other users have done this jceel/py-libzfs#18

[26O0]

edited the file and the few lines around it look like this:

        zfs = libzfs.ZFS(history=True, history_prefix="<iocage>")                                                                                                                                                                                                      
        pool = zfs.get(self.pool)                                                                                                                                                                                                                                      
        has_duplicates = len(list(filter(lambda x: x.mountpoint == u"/iocage",                                                                                                                                                                                         
                                         list(pool.root.datasets)))) > 0  

pyc file doesn't exist.

pip3 install -U . results in this:

   generating cffi module 'build/temp.freebsd-12.0-CURRENT-amd64-3.6/pygit2._libgit2.c'
    creating build/temp.freebsd-12.0-CURRENT-amd64-3.6
    building '_pygit2' extension
    creating build/temp.freebsd-12.0-CURRENT-amd64-3.6/src
    cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fstack-protector -fno-strict-aliasing -fPIC -I/usr/local/include -I/usr/local/include/python3.6m -c src/blob.c -o build/temp.freebsd-12.0-CURRENT-amd64-3.6/src/blob.o
    In file included from src/blob.c:30:
    In file included from src/blob.h:34:
    src/types.h:36:2: error: You need a compatible libgit2 version (v0.24.x)
    #error You need a compatible libgit2 version (v0.24.x)
     ^
    1 error generated.
    error: command 'cc' failed with exit status 1
    
    ----------------------------------------
  Rolling back uninstall of pygit2

My make looks like the output in that issue you linked. I don't see a solution, though.

[skarekrow]

Yeah that's because you have the newer pygit2 installed, you can manually remove /usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py and copy that file over to /usr/local/lib/python3.6/site-packages/iocage/lib/ioc_check.py in its place.

And yes, it seems no solution was suggested yet. I have very little Cython experience, so I'm not the best one to fix that error sadly :( Otherwise I'd send him a PR to unblock you three. Perhaps @william-gr has the ability to do so.

[26O0]

@skarekrow, those look like the same path.
Would I remove from /usr/local and copy from iocage src back to /usr/local?

[skarekrow]

Yes, sorry I didn't make that more clear.

EDIT: Just to be more clear, make sure it's just the ioc_check.py file that you are removing and copying over, not the rest.

[26O0]

make is still failing and so is pip3 install -U .

Looks like we'll need to wait and see what happens on jceel/py-libzfs#18

[skarekrow]

Yeah I believe that's blocking any meaningful progress here, and the pip3 will continue to fail until I update iocage's setup.py for the latest port, which isn't up to date on HardenedBSD yet, so I haven't done it :)

I'll leave this open and see what some of the guys I pinged have to say, I would suggest in the mean time rolling back to the 0.9.7 release unless you needed one of the many fixes and additions 0.9.8 brought.

Thanks for the issue!

[26O0]

@skarekrow, thanks for spending so much time with me tracking this down.

I'll download 0.9.7 and use that for the time being!

Thanks again, sir.

[skarekrow]

@26O0 and thank you for also spending that time trying to narrow things down. It's too bad we had to leave it in this state, though thinking about it again (man it is late...) you may not be able to use 0.9.7 either since the same libzfs issue will be a blocker as well :(

Hopefully we can get this issue solved sooner than later!

[26O0]

great! I'll await for the next trueOS build to try building iocage from git source.

[jungle-boogie]

I'm having this problem, and I'm pretty sure I have the latest devel/py-libzfs: py27-libzfs-1.0.20170525

[skarekrow]

@william-gr is the port maintainer and didn't bring forward the patch that was done as he felt there is a better way to fix it. But he couldn't reproduce the problem. So I'll ping him here and if you could @jungle-boogie please post something he can reproduce.

[william-gr]

The patch was not correct, it simply did masquerade the issue. There is a race condition in py-libzfs.

[jungle-boogie]

hi @william-gr, so you'll update py-libzfs on freebsd ports and @kmoore134 will pick it up and patch it again?

[william-gr]

There is no fix for py-libzfs yet. I am not the original author and could not reliably reproduce it to try and fix.

[jceel]

Why don't you guys open a new issue at https://github.com/jceel/py-libzfs/issues? or submit a pull request? it's the only way to make sure your local patches are correct way to solve problem.

[asomers]

I'm running into this with the latest TrueOS with iocage 0.9.9.1 and libzfs 1.0.20170814. Is there any workaround?

[skarekrow]

@araujobsd any ideas ^

[araujobsd]

I didn't check it yet, I know it is a long time issue! It is in my TODO list and I'll check it soon.

[asomers]

For now I reapplied the old patch at https://github.com/trueos/freebsd-ports/commit/00da370342012d87331eae3d74ef6784ed8172be and I can use iocage again.

[araujobsd]

@asomers Thanks for the feedback, I will check probably next week if that patch is the right thing to fix it or at least can mitigate while we hunt the real bug.

[disasmwinnie]

I get the same compiler errors like @26O0. I think py-libzfs generally doesn't work with freebsd 11.1 (or dereviates) since, the libzfs-API changed. I opened an upstream issue with some documentation: truenas/py-libzfs#1

However, I am not sure if the API change is responsible for all the issues mentioned before. I think I found the issue, but no solution. Hope somebody with knowledge about libzfs could have a lot at the problem.

[skarekrow]

8d2cd72 should fix this