add more developer docs #1089
add more developer docs #1089
Conversation
|
The steps I followed: I have the following deb lines in the sources.lsit Then installed the following packages: With these I can see debug symbols in the backstrace. Example: The command used: While creating a virtualenv for |
sssoleileraaa
force-pushed
the
developer-docs-2
branch
2 times, most recently
from
eca632b
to
dfaabb4
Compare
eloquence
moved this from Under Review
to Sprint #52 - 6/3-6/17
in SecureDrop Team Board
README.md
Outdated
| Sometimes there is a bug in Qt rather than the client, so it helps to install [a debug version of PyQt5](#build-and-install-a-debug-version-of-pyqt5) which will allow you to see Qt debug symbols in your Python tracebacks. You can use `gdb` to view a traceback as well as a core file, if you're debugging a segfault, but first you must need to install the following some additional packages in order for `gdb` to work with Python programs. | ||
|
|
||
| ``` | ||
| sudo apt install libc6-dbg libpython3-all-dbg libpython3-dbg libpython3.7-dbg python3-dbg python3.7-dbg valgrind-dbg |
There was a problem hiding this comment.
We should add the other dependency installation steps here. In my fresh Debian Qt refused to build due to missing dependencies.
There was a problem hiding this comment.
This list includes the dependencies for gdb to work with Python programs. It sounds like you had issues during the next step to clone the Qt repo and build it from source following https://wiki.qt.io/Building_Qt_5_from_Git#Getting_the_source_code, which defaults to creating a 'debug' build and installs the binaries in the current directory:
git clone https://code.qt.io/qt/qt5.git
cd qt5
git checkout v5.14.2 # or checkout a different version
git submodule update --init --recursive
export LLVM_INSTALL_DIR=/usr/llvm
mkdir ../qt5-build && cd ../qt5-build # we don't want to build in the source code directory
../qt5/configure -developer-build -opensource -nomake examples -nomake tests -confirm-license
make -j 4
Could you paste the output of the error you're seeing? And just to make sure you followed the instructions from https://wiki.qt.io/Building_Qt_5_from_Git#Getting_the_source_code and the steps I pasted above?
|
|
||
| ### Build and install a debug version of PyQt5 | ||
|
|
||
| 1. Build a debug version of Qt 5 |
There was a problem hiding this comment.
We should also add a line about required free space for a debug build.
There was a problem hiding this comment.
good call.. my qt5-build directory is 1.9 GB, is that what you're seeing?
There was a problem hiding this comment.
I am seeing significantly more than that:
user@dev-python:~$ du -sh qt5*
6.7G qt5
5.2G qt5-build
|
@emkll has committed to taking a spin through these docs as part of the 7/23-8/5 sprint. |
There was a problem hiding this comment.
Thanks @creviera the docs are very clear, added a couple of comments/suggestions inline.
I have successfully managed to build both Qt and PyQt using these instructions. However ran into a runtime issue when attempting to run the client with the debug version of PyQT:
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/user/securedrop-client/securedrop_client/__main__.py", line 1, in <module>
from .app import run
File "/home/user/securedrop-client/securedrop_client/app.py", line 30, in <module>
from PyQt5.QtCore import Qt, QTimer
RuntimeError: the sip module implements API v12.0 to v12.7 but the PyQt5.QtCore module requires API v12.8
After updating pyqt-sip to the latest version (12.8.0 from 4.19.19) I am getting the following when trying to run the client:
(.venv) user@dev-python:~/securedrop-client$ ./run.sh
Running app with home directory: /tmp/tmp.VFkg6JHi5E
Detected QubesOS, enabling DispVMs for submission handling...
gpg: keybox '/tmp/tmp.VFkg6JHi5E/gpg/pubring.kbx' created
gpg: /tmp/tmp.VFkg6JHi5E/gpg/trustdb.gpg: trustdb created
gpg: key CC40EF1228271441: public key "SecureDrop Test/Development (DO NOT USE IN PRODUCTION)" imported
gpg: key CC40EF1228271441: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: linuxfb, minimal, offscreen, vnc.
./run.sh: line 64: 22075 Aborted python -m securedrop_client --sdc-home "$SDC_HOME" --no-proxy "$qubes_flag" "$@"
Looks like we missing a Qt platform plugin. Based on this thread [1], seems like there's a configure option to force building libxcb.so, so will try this next.
[1] : https://forum.qt.io/topic/115827/build-on-linux-qt-xcb-option/16
| export LLVM_INSTALL_DIR=/usr/llvm | ||
| mkdir ../qt5-build && cd ../qt5-build # we don't want to build in the source code directory | ||
| ../qt5/configure -developer-build -opensource -nomake examples -nomake tests -confirm-license | ||
| make -j 4 |
There was a problem hiding this comment.
This step took ~1h on my computer, perhaps worth mentioning either here or elsewhere in the docs that this takes a while
There was a problem hiding this comment.
We will also need to build qttools here for the PySide2.
make -j 4 module-qttools
README.md
Outdated
| Sometimes there is a bug in Qt rather than the client, so it helps to install [a debug version of PyQt5](#build-and-install-a-debug-version-of-pyqt5) which will allow you to see Qt debug symbols in your Python tracebacks. You can use `gdb` to view a traceback as well as a core file, if you're debugging a segfault, but first you must need to install the following some additional packages in order for `gdb` to work with Python programs. | ||
|
|
||
| ``` | ||
| sudo apt install libc6-dbg libpython3-all-dbg libpython3-dbg libpython3.7-dbg python3-dbg python3.7-dbg valgrind-dbg |
There was a problem hiding this comment.
The following additional packages were required (to get configure to not output ERROR on configuration):
sudo apt-get install libclang-dev '^libxcb.*-dev' libx11-xcb-dev libglu1-mesa-dev libxrender-dev libxi-dev libxkbcommon-dev libxkbcommon-x11-dev gperf bison flex libgl1-mesa-dev pkg-config libdbus-1-dev libnss3-dev libfontconfig1-dev libxi-dev libxcomposite-dev libxcursor-dev libxtst-dev
|
|
||
| ### Build and install a debug version of PyQt5 | ||
|
|
||
| 1. Build a debug version of Qt 5 |
There was a problem hiding this comment.
I am seeing significantly more than that:
user@dev-python:~$ du -sh qt5*
6.7G qt5
5.2G qt5-build
| export PATH=~/qt/qt5-build/qtbase/bin:$PATH | ||
| export QT_PLUGIN_PATH=~/qt/qt5-build/qtbase/plugins | ||
| sip-install --debug --qmake ~/qt/qt5-build/qtbase/bin/qmake --confirm-license | ||
| ``` |
There was a problem hiding this comment.
This step took 15-20 minutes (17M of disk space used, which is probably negligible in context)
There was a problem hiding this comment.
Quick update on some findings:
I tried again using the following flags at the configure step to have xcb plugin support with -xcb -xvb-xlib , but compilation fails in both cases at some point during the process without any helpful error. I have also attempted to specify system xcb support by using the -xcb -system-xcb configuration flags (per https://forum.qt.io/post/600712, states that distribution-provided xcb may be needed since it is no longer shipped with Qt ), but also resulting in a failed compilation.
It may be worth attempting on another platform to see if the xcb requirement is platform specific, but we will likely need this support to be able to debug PyQt in Qubes
README.md
Outdated
| tar -xzvf PyQt5-5.14.2.tar.gz | ||
| cd PyQt5-5.14.2 | ||
| pip install PyQt-builder | ||
| export PATH=~/qt/qt5-build/qtbase/bin:$PATH |
There was a problem hiding this comment.
in https://github.com/freedomofpress/securedrop-client/pull/1089/files#diff-04c6e90faac2675aa89e2176d2eec7d8R349, the qt5-build folder is at the same level as the cloned repo qt5. Does this here refer to another folder inside the qt5 clone repository? if so, i don't see it myself.
|
Per discussion at backlog review we may want to remove
and keep that issue open as a tracking issue until we're running a version of Qt which has a bugfix for the upstream issue it references. |
|
Switched wording to "towards" and added an update on #273 |
eloquence
moved this from In Development
to Near Term - SD Workstation
in SecureDrop Team Board
|
Very close but a bit more testing required, we've agreed we will pick this up again in a future sprint. |
eloquence
moved this from Near Term - SD Workstation
to Next sprint candidates
in SecureDrop Team Board
eloquence
moved this from Next sprint candidates
to Near Term - SD Workstation
in SecureDrop Team Board
eloquence
moved this from Near Term - SD Workstation
to Next sprint candidates
in SecureDrop Team Board
eloquence
moved this from Next sprint candidates
to Near Term - SD Workstation
in SecureDrop Team Board
emkll
force-pushed
the
developer-docs-2
branch
from
dfaabb4
to
1aa60de
Compare
|
This should now be ready for another review. I've updated the documentation here to get a debug build of the version of PyQT we are targeting, but I am not quite sure what to do next. @kushaldas @creviera could you take a look an append one commit to explain how to validate the build and how to use the debug version in development context? Note that the documentation was updated to build (Py)QT 5.11.3, the version that is being tracked in the requirements files as well as Debian Buster. Given the reliance on system libraries for the build, it may be very difficult to target other versions of Qt/PyQt without manually installing some libraries/headers/dependencies, like sip and others. |
emkll
moved this from Near Term - SD Workstation
to Ready for Review
in SecureDrop Team Board
emkll
force-pushed
the
developer-docs-2
branch
from
c636dde
to
da9f62d
Compare
Update instructions for Qt 5.11.3
Matches version of python3-sip-dev in debian-10 (which includes headers required for pyqt build)
emkll
force-pushed
the
developer-docs-2
branch
from
da9f62d
to
b2cf872
Compare
|
We may want to also add a "how to build pyqt wheel" section, see #1165 (comment) |
|
The following is to build Setup CLANGGet the sourceCreate the virtualenvModify path and build the wheelPlease modify the following variables properly to the correct path in your system. For my setup, everything is under After this, you can find three wheels inside of the |
eloquence
moved this from In Development
to Near Term - SD Workstation
in SecureDrop Team Board
|
@kushaldas I followed the instructions above and run into this error when I get to the build step: It looks like the expected CMake file is missing. I'll look into this more tomorrow but maybe you can see what I'm doing wrong here? |
|
closing until we pick this back up again |
Description
Towards #273
Closes #1084
Add more instructions on how to debug and develop the client.