Posix shell: rebased & ready #10173

squashed & rebased version, not including: - listener changes - src/firmware renaming Commits: tag_to_version.py: fix Python3 error subprocess.communicate returns bytes instead of a str which is not the same for Python3. Therefore, we need to decode the bytes. cmake: remove folder src/firmware The folder src/firmware was not intuitive. Why would the binaries for SITL be inside a src and why even inside a src/firmware folder. Also, the rootfs was put there which made it even more confusing. The CMakeLists.txt files are moved into cmake/ and get now called from the main CMakeLists.txt. qshell: support for return value Instead of just sending commands, qshell will now also wait until the command has finished on QURT and sent back a return value. This will allow all modules on the DSP side to be spawned from the Linux side meaning that we only need one config/startup file instead of two. adb_upload: create folders before pushing Previously the script failed if the folder on the destination was not already existing. This therefore makes pushing easier. posix: spawn PX4 modules in bash This adds the possibility to spawn PX4 modules out of bash. Basically, the main executable can now be started as a server/daemon or as a client. The server replaces the existing functionality of the main exe with the pxh shell, however, it also opens a pipe that clients can talk to. Clients can run or spawn PX4 modules or commands by connecting to the server over the pipe. They clients will get the stdout and return value of their commands via a client specific pipe back. This work will allow to start all modules using a bash script similar to the way it is done in NuttX where the NuttShell scripts the startup scripts and starts the modules. SITL: use new client shell in SITL This is a first step to use the new shell capabilities for SITL. The new startup bash script rcS merges (and therefore replaces) the two existing scripts rcS_gazebo_iris and rcS_jmavsim_iris. More cleanup will be necessary for the rest of the SITL startup scripts. Snapdragon: use new shell to start all modules Instead of different mainapp.config and px4.config files, we can now use a unified rcS bash script which starts all the modules based on parameters, mainly the SYS_AUTOSTART param. Snapdragon: fix the airframe description pxh: argv needs to end with a nullptr The comment was wrong that argv needs an additional 0 termination. Instead it needs a nullptr at the end. px4_posix_tasks: variable cleanup The px4_task_spawn_cmd function got a cleanup while debugging, however, no functional changes. Snapdragon: move some drivers to 4100 config These drivers are supported by the community, so they go into the 4100 config. Snapdragon: update 210qc platform px4_daemon: use doxygen comments apps.h_in: fix string printf: use .c_str() px4_daemon: \b -> \n in printf px4_daemon: handle error in generate_uuid (close the file on error) posix main: some clarifications in comment (it's the symlinks not the script aliases) cmake: remove new install command again This one was probably wrong and untested. Installing needs revisiting. POSIX: remove argument USES_TERMINAL POSIX: copy init and mixer files for SITL Instead of using non-working install commands, the mixer and startup files are now copied as part of the build in cmake. adb_upload.sh: remove leftover commented printf POSIX main: just the pointer instead of memmove POSIX main: remove chroot chroot is removed because it hasn't been used anywhere and seems untested. px4_daemon: remove client pipe when cleaning up px4_daemon: fail if the client pipe already exists The client pipe is supposed to be specific (by UUID), so the path shouldn't exist already. history: limit the number of history entries This is a protection to avoid filling the memory if we are entering a lot of commands (e.g. auto-generated). px4_daemon: add a threadsafe map and use it px4_daemon: whitespace px4_daemon: fix client parsing Sometimes the client ends up reading more than one packet in one read. The parsing is not made for this and would require a (ring)buffer for it. The solution of this commit just reads as much as needed from the pipe which avoids having to do buffering and parsing. posix: changes sitl_run.sh and main.cpp cleanup This changes the paths in sitl_run.sh quite a bit to allow the px4 binary to run in the rootfs directory which should make it convenient and very close to the NuttX variant. Also main.cpp got a big cleanup after the big rebase with some conflicts. Quite some functionality was removed but it has yet to be seen if it needs to be re-added. px4_log: cleanup log levels, now they make sense Before DEBUG and INFO log levels where inverted which didn't make much sense in my eyes. dataman: fix path for bash shell logger: fix paths for bash shell mavlink: fix paths for bash shell param: fix path for bash shell inav: fix paths for bash shell sdlog2: fix paths for bash shell ROMFS: add forgotten mixer to list SITL init: more models, more options - Support for different models using the unified startup script rcS. - Support to choose the estimator by setting the environment variable PX4_ESTIMATOR. - Support to choose the logger by setting the environment variable PX4_LOGGER. rcS: fix string comparison listener: use template file Instead of having all of the C++ code inside the Python file it is nicer to have a separate template file with the C++ headers, etc. px4_log: add PX4_INFO_RAW for raw printfs This allows to do custom formatting but is still transported over sockets to clients. topic_listener: use PX4_INFO_RAW instead of printf commander: use PX4_INFO_RAW for status listener: rewrite to classes and factory posix: fix some argument warnings generate_listener.py: by accident changed shebang listener: big refactor of the generator Hopefully this makes it easier to read and change in the future. rcS: manually take over rebase changes listener: remove leftover try listener: properly clean up topic instance rcS: take over some vehicle specific changes posix-configs: vehicle specifics to separate files posix-configs: remove leftover lines uORBDevices: new PX4_INFO_RAW instead of printf px4_log: just use printf on NuttX listener: use less binary space, strip on NuttX generate_listener.py: remove commented code cmake: fix syntax error from merge px4_daemon: fixes after rebase of apps.h/cpp fix px4_daemon: namespace missing posix: only create stub for fsync on QURT unitests: reduce dependencies of param test This makes the unit test compile and link again after the bash changes. QURT: some compile fixes after a rebase SITL: arg change for sitl_run.sh to use rcS_test This allows to use a custom startup file for testing. SITL: add the folder test_data SITL: implement shutdown command as systemcmd The shutdown command needs to be a proper systemcmd, otherwise the alias and symlink generation doesn't work and we end up calling shutdown of the host computer which is to be avoided. px4fmu_test: same IO_pass mixer as px4fmu_default px4fmu_test: use normal quad x mixer There is no good reason to use a specific test mixer, except more cmake code around it. Therefore just use the same mixer as default, and at some point px4fmu_test and px4fmu_default can get merged POSIX: cleanup, dir and symlink fixes This cleans up the logic behind the symlinking and creating directories. POSIX: correct arg order in usage info tests: fix paths for SITL tests POSIX: printf fix sitl_run.sh: try to make this run on Mac as well cmake: try to make jenkins happier Path cleanup, the bin is no longer in src/firmware POSIX: fix symlink logic SITL: prefix all exported env variables cmake: fix path for ROS tests integrationtests: fix log path launch: try to make tets with ROS working again px4_defines: fix after wrong merge deconflicting px4_defines: get paths for POSIX correct cmake: fix cmake arguments This was fine with cmake 3.6 but did not work with cmake 3.2.2 cmake: use cp instead of cmake -E copy cmake -E copy does not support copying multiple files with versions < 3.5. Therefore, just use cp for now. ROMFS: fix build error after rebase cmake: fix paths in configs launch: use `spawn_model` again cmake: various fixes after big rebase param: path fixes after rebase posix platform: fixes after rebase test_mixer: fix screwed up rebase

- argv[0] can have the form ./px4-commander - create eeprom directory - fix fpe signal handler

- move posix-configs/SITL/init/{rcS,10016_iris,6011_typhoon_h480} to ROMFS/px4fmu_common/init.d-posix allows for easier unification, only one symlink is required. - rcS: add AUTOCNF support. Update scripts to match with behavior of PX4 master (parameter values, some sitl driver got removed) - add alias to allow 'set variable value' syntax in scripts to px4-alias.sh - use px4_getopt - use separate argument for the test_data directory - append PATH from within the px4 binary: this simplifies the usage w/o the sitl_run.sh script. - add 'source px4-alias.sh' to all existing sitl startup scripts - move sitl mixers to ROMFS/px4fmu_common/mixers-sitl makes it easier to use existing mixers and sitl-specific ones. - remove unused rcS_gazebo_delta_wing

This was printed for all submodules.

- RPi works - the rest is not tested, and requires a bit more (such as uploading the bin/ directory)

The buffer length given to snprintf() is unsigned, so we must check for an overflow after each call. This happend for very long printf's (module documentation)

The enum change from uint8_t to an int avoids uninitialized bytes, which led to valgrind warnings (no error though).

It was not enough to lock individual accesses to the maps. For example it could happen that a thread was started and exited very quickly, before the pthread_t item was inserted into the map, such that when the cleanup method was called, the thread and pipe fd were not found (and fd=0=stdin was closed).

By reading sizeof(packet), it could have happened that 2 packets at once were read, and the second one was then ignored. This makes sure that only one packet at a time is read. The other 2 changes are for efficiency.

Ensures that clients will get the output of the listener.

Also ignore the warning: error: calling 'system' uses a command processor [cert-env33-c,-warnings-as-errors]

This is to not lose any existing functionality until we have everything unified.

…enerate_symlinks

Move ros logs dir so that .ulg files are not in the same directory (mission_test.py:: get_last_log() fails otherwise)

Otherwise these files are added to the NuttX ROMFS build.

…d px4-alias.sh)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Posix shell: rebased & ready #10173

Posix shell: rebased & ready #10173

Commits on Aug 8, 2018