dist/tools/compile_test: Add Python3 support

[jnohlgard]

Tested working in Python 2.7.12 and Python 3.4.5

Python 3 is becoming more widely supported by distributions, so it is wise to stay up to date with our scripts so that they continue to run in the future.

[jnohlgard]

I know this is a bit ugly and makes it less readable, but it was the only way I could get it to work in both Python 2.7 and Python 3.x without resorting to writing two separate if branches.

[miri64]

Not less readable as before IMHO ^^.

[miri64]

Fine with the changes. ACK.

[miri64]

Not less readable as before IMHO ^^.

[miri64]

Can you squash, please?

[jnohlgard]

Found an error with Python3 when counting the number or errors and warnings. Fixed by explicitly constructing a list from the iterator produced by the filter function. (Python2 filter produces a list directly)

[jnohlgard]

Fixed a mistake when using Python3 and printing the errors, immediately squashed. (Added the list constructor on the three xxx_with_output lines)

[miri64]

Somehow Murdock does not want to start building. @kaspar030 any idea?

[kaspar030]

Somehow Murdock does not want to start building. @kaspar030 any idea?

There's a concurrency bug in murdock which makes it get stuck sometimes. Happened two or three times in the last six months, and I don't have an easy way to trigger it. I've now added a little more conservative locking and hope it will help.

[miri64]

Something went wrong. Murdock reports fails where there are none :-/

[jnohlgard]

oops, Python exception in unittests. I'll investigate later

[miri64]

@gebart ping?

[miri64]

Postponed due to feature freeze

[jnohlgard]

I am trying to debug the Python unicode exceptions via Murdock because I haven't been able to reproduce them locally.

[kaspar030]

Somehow the buildgroups get stuck. ps output looks like this:

riot-ci   3898  0.0  0.0  18052  2232 ?        S    00:13   0:00                  \_ /bin/bash bin/build_all.sh
riot-ci   3931  0.0  0.0  18060  2764 ?        S    00:13   0:00                      \_ bash ./dist/tools/ci/build_and_test.sh
riot-ci   3941  0.0  0.0  22016  7036 ?        S    00:13   0:00                          \_ python ./dist/tools/compile_test/compile_test.py master
riot-ci   4057  0.0  0.0   7588  2904 ?        S    00:13   0:00                              \_ make buildtest
riot-ci   5806  0.0  0.0   4532  1740 ?        S    00:13   0:00                                  \_ /bin/sh -c \ BUILDTESTOK=true; \ APP_RETRY=0; \ for BOARD in $(make -s info-boards-supported); do \   RIOTNOLINK=$(echo airfy-beacon cc2650stk chronos ek-lm4f120xl msb-430 msb-430h pca10000 pca10005 spark-core stm32f0discovery stm32f3discovery telosb wsn430-v1_3b wsn430-v1_4 z1 nucleo-f103 nucleo-f334 yunjia-nrf51822 samr21-xpro arduino-mega2560 airfy-beacon nrf51dongle nrf6310 weio waspmote-pro nucleo-f072 arduino-uno arduino-duemilanove sodaq-autonomo arduino-zero nucleo-f030 nucleo-f070 nucleo-f091 pba-d-01-kw2x saml21-xpro microbit calliope-mini limifrog-v1 slwstk6220a | grep ${BOARD} 2>&1 >/dev/null && echo 1); \   /bin/echo -n "Building for ${BOARD} "; \   [ -n "${RIOTNOLINK}" ] && /bin/echo -n "(no linking) "; \   for NTH_TRY in 1 2 3; do \     /bin/echo -n ".. "; \     if [ "$NTH_TRY" != "3" ]; then export _CCACHE=$CCACHE; else export _CCACHE=""; fi ; \     LOG=$(env -i HOME=${HOME} PATH=${PATH} BOARD=${BOARD} CCACHE=${_CCACHE} ${CCACHE_DIR:+CCACHE_DIR=${CCACHE_DIR}} ${GIT_CACHE_DIR:+GIT_CACHE_DIR=${GIT_CACHE_DIR}} CCACHE_BASEDIR=${CCACHE_BASEDIR} RIOTBASE=${RIOTBASE} RIOTBOARD=${RIOTBOARD} RIOTCPU=${RIOTCPU} RIOTPKG=${RIOTPKG} BINDIRBASE=${BINDIRBASE} RIOTNOLINK=${RIOTNOLINK} RIOT_VERSION=${RIOT_VERSION} WERROR=${WERROR} LTO=${LTO} TOOLCHAIN=${TOOLCHAIN} make -j8 clean all 2>&1) ; \     if [ "${?}" = "0" ]; then \       /bin/echo "success"; \       if [ -n "${BUILDTEST_VERBOSE}" ]; then \         echo "${LOG}" | tail -n +2 | head -n -2 | grep -v -E '^Building application|^\"make|^patching' | awk 'NF'; \       fi; \     elif [ -n "${RIOT_DO_RETRY}" ] && [ "${APP_RETRY}" -lt "3" ] && [ ${NTH_TRY} != 3 ]; then \       /bin/echo -n "retrying "; \       continue; \     else \       /bin/echo "failed"; \       if [ -n "${BUILDTEST_VERBOSE}" ]; then \         echo "${LOG}" | grep -v -E '^\"make'; \       fi; \       APP_RETRY=`expr ${APP_RETRY} + 1`; \       BUILDTESTOK=false; \     fi; \     break; \   done; \   env -i \     HOME=${HOME} \     PATH=${PATH} \     BOARD=${BOARD} \     CCACHE=${CCACHE} \     ${CCACHE_DIR:+CCACHE_DIR=${CCACHE_DIR}} \     CCACHE_BASEDIR=${CCACHE_BASEDIR} \     ${GIT_CACHE_DIR:+GIT_CACHE_DIR=${GIT_CACHE_DIR}} \     RIOTBASE=${RIOTBASE} \     RIOTBOARD=${RIOTBOARD} \     RIOTCPU=${RIOTCPU} \     RIOTPKG=${RIOTPKG} \     BINDIRBASE=${BINDIRBASE} \     RIOTNOLINK=${RIOTNOLINK} \     RIOT_VERSION=${RIOT_VERSION} \     make clean-intermediates 2>&1 >/dev/null || true; \ done; \ ${BUILDTESTOK}
riot-ci   9757  0.0  0.0   7600   980 ?        S    00:14   0:00                                      \_ awk NF

[jnohlgard]

@kaspar030 the exception came from ISO8859-1 encoded ó in the output from some pkg/relic code preparation, unsure whether the hung task was also related to that. The exception went away when I added errors='replace' on the decode line, which will replace any non-UTF8 letters by a special character (0xfffd) instead of raising an exception.

Updated commit by adding errors='replace', immediately squashed

[kaspar030]

Looks fine. ACK if Murdock agrees!