Mklauncher incompatible with python3

[the-snowwhite]

Mklauncher needs these fixes before even being able to run under python
However then running mklauncher --debug results in ending with following message (I figured out to also set command to python3 run.py in the configs launcher.ini) :-)

DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/mklauncher", line 398, in _poll
    self._update_launcher_status()
  File "/usr/bin/mklauncher", line 352, in _update_launcher_status
    stdout_line.line = line
TypeError: None has type NoneType, but expected one of: bytes, unicode

Also when I open a separate console and input:

~pgrep mklaucher~ 

When mklauncher is started up:

running in python2 gives the pid (process id)

running in python3 gives nothing (no process id)

Edit: due to typo correct command is:

pgrep mklauncher

I have no idea whats to the missing process id in python3 (leading to the config launcher subprocess attempting to restart the master mklauncher process thats initiating this subprocess)

Edit: This seems to be happening after the first run where the process is renamed to python3 on consecutive runs (fixed this via adding setproctitle('mklauncher'))
however the typeerror remains:

[the-snowwhite]

#329 (comment)

[the-snowwhite]

With some added debug info and commenting out of lines that don't seem to matter its possible to see the difference of the:
Working python 2 version:

DEBUG:mklauncher:process command called, id: ['\x00k\x8bEg']
DEBUG:mklauncher:process launcher called launcher True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_FULL_UPDATE
DEBUG:mklauncher:process command called, id: ['\x00k\x8bEg']
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:POLL: sent pb.MT_PING
DEBUG:mklauncher:process launcher called launcher False

and non functional python3 version:

DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
DEBUG:mklauncher:process launcher called launcher True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_FULL_UPDATE
DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
DEBUG:mklauncher:process launcher called launcher False
^CDEBUG:mklauncher:stopping threads

[the-snowwhite]

With this latest update
the Machinekit Client login shows similar logs for py2/3
py2:

DEBUG:mklauncher:process command called, id: ['\x00k\x8bEg']
DEBUG:mklauncher:process launcher called launcher True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_FULL_UPDATE
DEBUG:mklauncher:process command called, id: ['\x00k\x8bEg']
DEBUG:mklauncher:MSG: line = starting configserver... 
DEBUG:mklauncher:ulsMSG: if modified: --> has_update = True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:POLL: sent pb.MT_PING
DEBUG:mklauncher:MSG: line = done
starting linuxcnc... 
DEBUG:mklauncher:ulsMSG: if modified: --> has_update = True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:process launcher called launcher False

py3:

DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
DEBUG:mklauncher:process launcher called launcher True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_FULL_UPDATE
DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
DEBUG:mklauncher:ulsMSG: if modified: --> has_update = True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:MSG: line = starting configserver... 
DEBUG:mklauncher:ulsMSG: if modified: --> has_update = True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:POLL: sent pb.MT_PING
DEBUG:mklauncher:MSG: line = done

DEBUG:mklauncher:MSG: line = starting linuxcnc... 
DEBUG:mklauncher:ulsMSG: if modified: --> has_update = True
DEBUG:mklauncher:sending launcher message
DEBUG:mklauncher:ulsMSG: sent pb.MT_LAUNCHER_INCREMENTAL_UPDATE
DEBUG:mklauncher:process launcher called launcher False

However in the py3 version Machinekit Client does not initialize Cetus properly and shows:

Not connected 

@bottom left

Oddly enough a fully python3 updated AND Demo works properly with mklauncher py3 startup.
So this perhaps has something to do with:

launcher.start_process('linuxcnc ultr.ini')

???

[cerna]

@the-snowwhite,
to tell the truth, I have tried to run Python 3 mklauncher in Ubuntu 20.04 Focal Docker container installed from the latest packages from MAchinekit-HAL and I cannot see the issue:

mk@2ca882c8c169:/$ python3 /usr/bin/mklauncher --debug
DEBUG:mklauncher:announcing mklauncher
DEBUG:mklauncher:parsed launchers:

service: dsname = tcp://%(fqdn)s:38621 port = 38621 txtrec = ['dsn=tcp://%(fqdn)s:38621', 'uuid=2456f9fa-d010-42a5-a989-25751c1884f1', 'instance=ac9c850c-2770-11eb-af2b-0242ac110002', 'service=launcher'] name = Machinekit Launcher on %(fqdn)s
service: dsname = tcp://%(fqdn)s:43085 port = 43085 txtrec = ['dsn=tcp://%(fqdn)s:43085', 'uuid=2456f9fa-d010-42a5-a989-25751c1884f1', 'instance=ac9c8bec-2770-11eb-af2b-0242ac110002', 'service=launchercmd'] name = Launchercmd service on %(fqdn)s pid 9348

And I can see the service from the base system:

(After installing the DBus daemon and Avahi daemon in the Docker container. Of course.)

However, that's Python 3.8. So maybe on other versions it is causing problems.

[the-snowwhite]

@cerna I understand
I can do the same on my arm64 soc's
The problems begin when I attempt to run a qtquickvcp based config
like the one similar to this (for the Cetus gui).

mklauncher ~/Hm2-soc_FDM/Cramps/PY/FZ3 --debug 

I can run the Anddemo in python3
However the Anddemo only uses the halremote remote communication protocol.

The configs based on the Cetus gui (Probaly likewise for Machineface) use
a different communication style (Machinetalk ?)

I only have Cetus configs for the Hostmot2 socfpga on hand.

It might be best if we could to spin up a amd64 Cetus sim config as a common test reference ?

[the-snowwhite]

I managed to replicate your amd64 docker setup with a dockerfile.
@cerna
Please install the anddemo in your docker
run mklauncher anddemo --debug
and then attempt to login with the Machinekit Client
Then you will see the very first error message i had to fix ...

[cerna]

@the-snowwhite,
my Docker recipes (it's ugly because it is just quickly hacked version of Machinekit-HAL builder):

Dockerfile

FROM ubuntu:focal

SHELL [ "bash", "-c" ]

ENV TERM=dumb

ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
ENV LC_ALL=C.UTF-8
ENV LANGUAGE=C.UTF-8
ENV LANG=C.UTF-8

RUN printf "%s;\n%s;\n"                 \
    'APT::Install-Recommends "0"'       \
    'APT::Install-Suggests "0"'         \
    > /etc/apt/apt.conf.d/01norecommend

RUN apt-get update &&                                                                 \
    apt-get install -y                                                                \
        curl                                                                          \
        apt-transport-https                                                           \
        ca-certificates &&                                                            \
    curl -1sLf                                                                        \
    'https://dl.cloudsmith.io/public/machinekit/machinekit/cfg/setup/bash.deb.sh'     \
        | bash &&                                                                     \
    curl -1sLf                                                                        \
    'https://dl.cloudsmith.io/public/machinekit/machinekit-hal/cfg/setup/bash.deb.sh' \
        | bash &&                                                                     \
    apt-get clean

RUN apt-get update &&     \
    apt-get -y upgrade && \
    apt-get clean

ENV USER=machinekit

RUN addgroup --gid 1000 ${USER} &&                            \
    adduser --uid 1000 --ingroup ${USER} --home /home/${USER} \
    --shell /bin/bash --disabled-password --gecos "" ${USER}

RUN apt-get update &&        \
    apt-get install -y       \
        sudo                 \
        machinekit-fixuid && \
    apt-get clean

COPY base-entrypoint.sh /opt/bin/base-entrypoint.sh

RUN chmod +x /opt/bin/base-entrypoint.sh &&                       \
    mkdir /opt/environment &&                                     \
    echo "${USER} ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers && \
    mkdir -p /etc/fixuid &&                                       \
    printf "user: ${USER}\ngroup: ${USER}\n" > /etc/fixuid/config.yml

ENTRYPOINT [ "/opt/bin/base-entrypoint.sh" ]

RUN apt-get update &&      \
    apt-get install -y     \
        dbus               \
        avahi-daemon       \
        machinekit-hal     \
        machinekit-hal-dev \
        git &&             \
    apt-get clean;

RUN mkdir -p /var/run/dbus && \
    sed -i 's/REMOTE=0/REMOTE=1/' /etc/machinekit/machinekit.ini

And base-entrypoint.sh

#!/bin/bash

PASSED_USER_UID=$(id -u)
PASSED_USER_GID=$(id -g)
IMAGE_USER_UID=$(id -u ${USER})
IMAGE_USER_GID=$(id -g ${USER})

# Only allow running when user is specified and do not allow root user period
if (( PASSED_USER_UID == 0 || PASSED_USER_GID == 0 ))
then
    printf "%b"                                                                    \
           "Machinekit-HAL entrypoint: Running as the root is not allowed.\n"      \
           "Passed user ID:${PASSED_USER_UID} and group ID:${PASSED_USER_GID}.\n";
    exit 1
fi
# Catch cases when user pass -e USER to Docker run command
if ! [[ $PASSED_USER_UID =~ ^[0-9]+$ ]]
then
    printf "%b"                                                                 \
           "Machinekit-HAL entrypoint: Don not pass user ${USER} with -e.\n" \
           "EXITING.\n";                                                        \
    exit 1
fi

if (( PASSED_USER_UID == IMAGE_USER_UID &&
      PASSED_USER_GID == IMAGE_USER_GID ))
then
    printf "%b"                                                                \
           "Machinekit-HAL entrypoint: Nothing needs to be done.\n"            \
           "Passed user ID:${PASSED_USER_UID} and group ID:${PASSED_USER_GID}" \
           " is the same as UID and GID of IMAGE user ${USER}.\n"
else
    printf "%b"                                                                 \
           "Machinekit-HAL entrypoint: Changing burnt-in user ${USER} UID and/" \
           "or GID to ones passed by --user run option.\n"                      \
           "All files under / belonging to old user ID:${IMAGE_USER_UID} or"    \
           " group ID:${IMAGE_USER_GID} will be updated to new values of user"  \
           " ID:${PASSED_USER_UID} and group ID:${PASSED_USER_GID}.\n"

    (
        machinekit-fixuid
    )
fi

# Set up new HOME environment variable if nothing was passed to container
if [[ -z "${HOME}" || "${HOME}" == "/" ]]
then
    ENT="$(getent passwd "${USER}")"
    NEW_HOME=$(echo $ENT | cut -d : -f 6)
    export HOME=${NEW_HOME}
fi

# Prepare environment for process
ENVIRONMENT_DIRECTORY="/opt/environment"
mapfile -d '' -t ARRAY_ENV < <(find ${ENVIRONMENT_DIRECTORY} -type f -print0 \
    | xargs -0 -n1 -I '{}' realpath -e -z '{}')

for FILE in "${ARRAY_ENV[@]}"
do
    source ${FILE}
done

sudo dbus-daemon --config-file=/usr/share/dbus-1/system.conf --print-address
sudo service avahi-daemon restart

printf "ENTRYPOINT END\n======================================================\n"
exec "$@"

[cerna]

@the-snowwhite,

Please install the anddemo in your docker
run mklauncher anddemo --debug
and then attempt to login with the Machinekit Client
Then you will see the very first error message i had to fix ...

Yup, I can see it:

DEBUG:mklauncher:process command called, id: [b'\x00k\x8bEg']
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/mklauncher", line 263, in _process_sockets
    self._process_launcher_socket(self.launcher_socket)
  File "/usr/bin/mklauncher", line 422, in _process_launcher_socket
    ("process launcher called " + subscription + ' ' + str(status))
TypeError: can only concatenate str (not "bytes") to str

The configs based on the Cetus gui (Probaly likewise for Machineface) use
a different communication style (Machinetalk ?)

I have a feeling that the terminology was never really stabilized. Which is a damn shame, as concise nomenclature makes for understandable system.

It might be best if we could to spin up a amd64 Cetus sim config as a common test reference ?

From long term perspective, it would be very nice to have some form of integration test running in CI on timer. Otherwise, there is no chance of catching problems in multiple repositories with different development vectors.

[the-snowwhite]

The (mklauncher config) run.py script utilizes the pgrep mklauncher command to see if mklauncher is already running
and so I tested out if my arm64 findings of the process having the namepython3 instead were valid on a arm64 buster docker image after tripple checking thrice I found out that if you run mklauncher with:

python3 /usr/bin/mklauncher anddemo --debug

It also gets given the process name python3 rendering the pgrep test useless (this also counts for the Focal docker image).
These 2 docker images only have python3 installed.
Reason why my arm64 buster soc without the setproctitle fix consistently (95%) renames the process to python3
even when i just run:

mklauncher anddemo --debug

is perhaps because both python 2 and 3 are installed there. edit: python3-setproctitle package is installed

The setproctitle fix however adds another deb dependency: python3-setproctitle

@cerna

It might be best if we could to spin up a amd64 Cetus sim config as a common test reference ?

From long term perspective, it would be very nice to have some form of integration test running in CI on timer. Otherwise, there is no chance of catching problems in multiple repositories with different development vectors.

I was just thinking more short term:
To be able to work out the kinks of getting Cetus <> mklauncher <> Machinekit Client connect up and running ... :-)

[the-snowwhite]

I recon the mklauncher python3 puzzle as solved
The issues with Cetus/Machineface not connecting with Machineclient must lay elsewhere
judging by the now working anddemo.(via halremote)
I will redelegate the remaining to the mkwrapper repo thread and for now not recomend moving mklauncher
(since the Anddemo works fine without linuxcnc/emcapp)
@cerna this issue can be closed after merge of pr.