Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify Linux packaging-builds to include QtWebEngine #523

Closed
michaelsbradleyjr opened this issue Jul 7, 2020 · 24 comments · Fixed by #1126
Closed

Modify Linux packaging-builds to include QtWebEngine #523

michaelsbradleyjr opened this issue Jul 7, 2020 · 24 comments · Fixed by #1126
Assignees

Comments

@michaelsbradleyjr
Copy link
Contributor

The docker image used as the basis for building and packaging bin/nim_status_client is missing the QtWebEngine components, which results in a AppImage that lacks the browser.

For clarity's sake, what I mean is that even if a user enables it in Advanced Settings the browser tab will be blank because the needed software components are missing.

@jakubgs
Copy link
Member

jakubgs commented Oct 2, 2020

@jakubgs
Copy link
Member

jakubgs commented Oct 2, 2020

So we could fork it and adjust the available packages using QT_CI_PACKAGES env variable:

export QT_CI_PACKAGES=qt,qt.594,qt.594.gcc_64,qt.594.doc.qtvirtualkeyboard

@jakubgs
Copy link
Member

jakubgs commented Oct 2, 2020

Considering /tmp/build/install-qt.sh script is included in the image we use:

 > d run --rm -it a12e/docker-qt:5.14-gcc_64
user@4ff17343808b:~$ ls -l /tmp/build/install-qt.sh
-rwxr-xr-x 1 root root 874 Jan 11  2020 /tmp/build/install-qt.s

We could simply run it again in our own layer to adjust available packages with QT_CI_PACKAGES.
This way we can avoid forking the repo of the starting image we use.

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020

It looks like they are already setting QT_CI_PACKAGES inside of scripts/install-qt.sh:

QT_CI_PACKAGES=qt.qt5.$(echo "${QT_VERSION}" | tr -d .).${QT_PLATFORM} \

https://github.com/a12e/docker-qt/blob/851947208e090f3f62471e19d7b2bf8f8ca22dc8/scripts/install-qt.sh#L23

So we'd have to modify it using sed or something.

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020

The available packages are:

qt.qt5.5140
qt.qt5.5140.src
qt.qt5.5140.gcc_64
qt.qt5.5140.qtcharts
qt.qt5.5140.qtcharts.gcc_64
qt.qt5.5140.qtdatavis3d
qt.qt5.5140.qtdatavis3d.gcc_64
qt.qt5.5140.qtlottie
qt.qt5.5140.qtlottie.gcc_64
qt.qt5.5140.qtnetworkauth
qt.qt5.5140.qtnetworkauth.gcc_64
qt.qt5.5140.qtpurchasing
qt.qt5.5140.qtpurchasing.gcc_64
qt.qt5.5140.qtquick3d
qt.qt5.5140.qtquick3d.gcc_64
qt.qt5.5140.qtquicktimeline
qt.qt5.5140.qtquicktimeline.gcc_64
qt.qt5.5140.qtscript
qt.qt5.5140.qtscript.gcc_64
qt.qt5.5140.qtvirtualkeyboard
qt.qt5.5140.qtvirtualkeyboard.gcc_64
qt.qt5.5140.qtwaylandcompositor
qt.qt5.5140.qtwaylandcompositor.gcc_64
qt.qt5.5140.qtwebengine
qt.qt5.5140.qtwebengine.gcc_64
qt.qt5.5140.qtwebglplugin
qt.qt5.5140.qtwebglplugin.gcc_64

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020

I tried re-building the image by re-running the script like so:

Step 4/14 : RUN sudo sed -i 's/qt.qt5/qt.qt5.5140.qtwebengine.gcc_64,qt.qt5/' /tmp/build/install-qt.sh  && QT_VERSION=5.14.0 sudo -E /tmp/build/install-qt.sh
 ---> Running in f918b2d6e6c8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   271  100   271    0     0    768      0 --:--:-- --:--:-- --:--:--   769
100 1264M  100 1264M    0     0  6819k      0  0:03:09  0:03:09 --:--:-- 6431k

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'

But it just got stuck on that for more than 3 hours.

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020

I tried running it with VERBOSE=1 but it's not giving me anything useful, just gets stuck on:

[127] Loaded control script "/tmp/tmp.EJnnUB1h0"
[127] Using control script: "/tmp/tmp.EJnnUB1h0"
[144] QTCI:  Welcome Page

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020

Options:

  • Modify the original Dockerfile and adjust QT_CI_PACKAGES
  • Attempt to use the nix-build branch and try to switch the build from Docker image to Nix

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

A suggestion from @richard-ramos is to add export http_proxy=invalid to avoid getting stuck on Welcome Page as above.

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

I did it like this, adding export http_proxy=invalid to the Qt install script right before qtci is invoked:

# Re-install Qt with QtWebEngine by adjusting QT_CI_PACKAGES
# The http_proxy=invalid is a fix for getting stuck on 'Welcome Page'
RUN sudo rm -r $QT_PATH \
 && sudo sed -i 's/qt.qt5/qt.qt5.5140.qtwebengine.gcc_64,qt.qt5/' /tmp/build/install-qt.sh \
 && sudo sed -i '\#^/tmp/build/bin/extract-qt-installer.*#i export http_proxy=invalid' /tmp/build/install-qt.sh \
 && VERBOSE=1 QT_VERSION=5.14.0 sudo -E /tmp/build/install-qt.sh

And it works!

[149] Warning: Network error: [ QNetworkReply::NetworkError(HostNotFoundError) ] "Failed to connect to server. Check your network connection and try again."
[152] QTCI:  Introduction Page
[152] QTCI:  Retrieving meta information from remote repository
[158] Url is: "://metadata/"
[188] Warning: Component qt.qt5.5140 contains data to be installed while having child components. This may not work properly.
[188] Warning: Component qt.qt5.5140 depends on other components while having child components. This will not work properly.
[189] Warning: Other components depend on component qt.tools which has child components. This will not work properly.
[191] QTCI:  Set target installation page: /opt/qt
[195] QTCI:  ComponentSelectionPageCallback
[195] QTCI:  Select components
[198] QTCI:  Use default component list
[208] QTCI:  Accept license agreement
[216] Tmp and install directories are on the same volume. Volume mount point: "/" Free space available: "7.41 GB"
[216] Installation space required: "899.52 MB" Temporary space required: "256.00 MB" Local repository size: "0.00 bytes"
[216] QTCI:  Ready to install
[420] QTCI:  PerformInstallationPageCallback
[450] No extended license checking url found.
[451] backup  operation: Mkdir
[451] 	- arguments: /opt/qt
[451] Done
[451] perform  operation: Mkdir
[451] 	- arguments: /opt/qt
[451] Done
[451] Install size: 5 components
...
[11015] Done
[11017] Writing maintenance tool: "/opt/qt/MaintenanceTool.new"
[11059] Wrote permissions for maintenance tool.
[12855] Maintenance tool restart: false.
[12868] QTCI:  FinishedPageCallback
[12869] QTCI:  Press Finish Button
Cleaning Qt SDK: /opt/qt/components.xml
Cleaning Qt SDK: /opt/qt/dist
Cleaning Qt SDK: /opt/qt/Licenses
Cleaning Qt SDK: /opt/qt/Tools
Cleaning Qt SDK: /opt/qt/installer.dat
Cleaning Qt SDK: /opt/qt/MaintenanceTool.ini
Cleaning Qt SDK: /opt/qt/network.xml
Cleaning Qt SDK: /opt/qt/MaintenanceTool.dat
Cleaning Qt SDK: /opt/qt/MaintenanceTool
Cleaning Qt SDK: /opt/qt/InstallationLog.txt

Lovely.

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

I also learned that if you wan to use a different delimiter in sed for triggering an action like insert(i) you need to escape the first delimiter with \.

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

I tried without sudo rm -r $QT_PATH and it appears to have no effect on the final image size which is 1.93GB.

admin@linux-01.do-ams3.ci.misc:~ % d images statusteam/nim-status-client-build
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
statusteam/nim-status-client-build   latest              05a16db8f629        3 months ago        1.92GB

Slightly bigger from what we use right now, which makes sense.

jakubgs added a commit that referenced this issue Oct 6, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
richard-ramos pushed a commit that referenced this issue Oct 6, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

Something is wrong. I ran the build after removing /opt/qt and without modifying QT_CI_PACKAGES and I got nothing:

 > docker run -it --rm statusteam/nim-status-client-build:1.0.0 bash
cduser@fa6d9b260a63:~$ cd /opt/qt/
user@fa6d9b260a63:/opt/qt$ ll
total 0
drwxr-xr-x 2 root root  6 Oct  6 20:41 ./
drwxr-xr-x 3 root root 16 Oct  6 20:41 ../

Something is wrong with how Qt is installed.

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

The script we run removes some useless stuff after installation:

Cleaning Qt SDK: /opt/qt/components.xml
Cleaning Qt SDK: /opt/qt/dist
Cleaning Qt SDK: /opt/qt/Licenses
Cleaning Qt SDK: /opt/qt/Tools
Cleaning Qt SDK: /opt/qt/installer.dat
Cleaning Qt SDK: /opt/qt/MaintenanceTool.ini
Cleaning Qt SDK: /opt/qt/network.xml
Cleaning Qt SDK: /opt/qt/MaintenanceTool.dat
Cleaning Qt SDK: /opt/qt/MaintenanceTool
Cleaning Qt SDK: /opt/qt/InstallationLog.txt

But that should not affect the /opt/qt/5.14.0 folder.

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

More accurate list of packages:

qt
qt.qt5.5140
qt.tools
qt.license.win10sdk
qt.license.python
qt.license.lgpl
qt.installer.changelog
qt.license.gplv3except
qt.license.thirdparty
qt.qt5.5140.gcc_64
qt.qt5.5140.android
qt.qt5.5140.src
qt.qt5.5140.qtcharts
qt.qt5.5140.qtquick3d
qt.qt5.5140.qtdatavis3d
qt.qt5.5140.qtlottie
qt.qt5.5140.qtpurchasing
qt.qt5.5140.qtvirtualkeyboard
qt.qt5.5140.qtwaylandcompositor
qt.qt5.5140.qtwebengine
qt.qt5.5140.qtnetworkauth
qt.qt5.5140.qtwebglplugin
qt.qt5.5140.qtscript
qt.qt5.5140.qtquicktimeline
qt.qt5.5140.doc
qt.qt5.5140.examples
qt.qt5.5140.qtcharts.gcc_64
qt.qt5.5140.qtcharts.android
qt.qt5.5140.qtquick3d.gcc_64
qt.qt5.5140.qtdatavis3d.gcc_64
qt.qt5.5140.qtdatavis3d.android
qt.qt5.5140.qtlottie.android
qt.qt5.5140.qtlottie.gcc_64
qt.qt5.5140.qtpurchasing.gcc_64
qt.qt5.5140.qtpurchasing.android
qt.qt5.5140.qtvirtualkeyboard.gcc_64
qt.qt5.5140.qtwaylandcompositor.gcc_64
qt.qt5.5140.qtwebengine.gcc_64
qt.qt5.5140.qtnetworkauth.gcc_64
qt.qt5.5140.qtnetworkauth.android
qt.qt5.5140.qtwebglplugin.gcc_64
qt.qt5.5140.qtscript.android
qt.qt5.5140.qtscript.gcc_64
qt.qt5.5140.qtquicktimeline.gcc_64
qt.qt5.5140.qtquicktimeline.android
qt.qt5.5140.doc.qtscript
qt.qt5.5140.doc.qtvirtualkeyboard
qt.qt5.5140.doc.qtquick3d
qt.qt5.5140.doc.qtwebengine
qt.qt5.5140.doc.qtlottie
qt.qt5.5140.doc.qtnetworkauth
qt.qt5.5140.doc.qtdatavis3d
qt.qt5.5140.doc.qtpurchasing
qt.qt5.5140.doc.qtcharts
qt.qt5.5140.doc.qtquicktimeline
qt.qt5.5140.examples.qtquick3d
qt.qt5.5140.examples.qtquicktimeline
qt.qt5.5140.examples.qtcharts
qt.qt5.5140.examples.qtwebengine
qt.qt5.5140.examples.qtpurchasing
qt.qt5.5140.examples.qtnetworkauth
qt.qt5.5140.examples.qtvirtualkeyboard
qt.qt5.5140.examples.qtscript
qt.qt5.5140.examples.qtlottie
qt.qt5.5140.examples.qtdatavis3d
qt.tools.qtcreator

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

It looks like the selection needs to include qt and qt5.5140, as shown in this example:

export QT_CI_PACKAGES=qt,qt5.5140,qt.qt5.5140.gcc_64,qt.qt5.5140.qtwebengine

Which results in a proper installation which includes QtWebEngine:

user@3142ceebb6e3:/opt/qt/5.14.0/gcc_64/qml$ ls -l | grep QtWebEngine 
drwxr-xr-x  4 root root   127 Oct  6 21:01 QtWebEngine

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

Okay, this looks better:

 > docker run -it --rm statusteam/nim-status-client-build:1.0.0 bash 
jenkins@9790511624e7:/home/user$ ls -l /opt/qt/5.14.0/gcc_64/qml/ | grep QtWebEngine
drwxr-xr-x  4 root root   127 Oct  6 21:16 QtWebEngine

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

But daaaamn look at the size of the image:

REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
statusteam/nim-status-client-build   1.0.0               25dcc280dd29        12 minutes ago      5.22GB

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

It appears by using:

export QT_CI_PACKAGES=qt,qt5.5140,qt.qt5.5140.gcc_64,qt.qt5.5140.qtwebengine

I'm also including other stuff:

jenkins@45557267cd56:/opt/qt/5.14.0$ du -hsc ./*
2.9G	./Src
620M	./android
778M	./gcc_64
4.0K	./sha1s.txt
4.3G	total

@jakubgs
Copy link
Member

jakubgs commented Oct 6, 2020

That's better:

export QT_CI_PACKAGES=qt.qt5.5140.gcc_64,qt.qt5.5140.qtwebengine,qt.qt5
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
statusteam/nim-status-client-build   1.0.0               194ec1b9d22a        13 seconds ago      2.09GB

I think there was a bit of a red herring due to \ at the end of line in that script which was broken by addition of the other export.

jakubgs added a commit that referenced this issue Oct 6, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
richard-ramos pushed a commit that referenced this issue Oct 6, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@jakubgs
Copy link
Member

jakubgs commented Oct 7, 2020

I've updated #1126 with adjusted QT_CI_PACKAGES and it looks correct now:

# Re-install Qt with QtWebEngine by adjusting QT_CI_PACKAGES
# The http_proxy=invalid is a fix for getting stuck on 'Welcome Page'
RUN sudo sed /tmp/build/install-qt.sh -i \
      -e 's/^QT_CI_PACKAGES=.*/export QT_CI_PACKAGES=qt.qt5.5140.gcc_64,qt.qt5.5140.qtwebengine,qt.qt5/' \
      -e '\#^/tmp/build/bin/extract-qt-installer.*#i export http_proxy=invalid' \
 && sudo -E /tmp/build/install-qt.sh

@richard-ramos can you please test it with the browser branch when you have some time?

@richard-ramos
Copy link
Member

Did a quick test with #1128
linuxdeployqt is idenitifying the webengine qml component! althrough the packaging step is failing due to not finding a shared object: libsmime3.so which is required by it

[2020-10-07T12:16:24.782Z] Building: AppImage
[2020-10-07T12:16:24.782Z] linuxdeployqt 6 (commit 5195ec1), build 712 built on 2019-11-22 21:06:09 UTC
[2020-10-07T12:16:24.782Z] Desktop file as first argument: "tmp/linux/dist/nim-status.desktop"
[2020-10-07T12:16:24.782Z] desktopExecEntry: "nim_status_client"
[2020-10-07T12:16:24.782Z] desktopIconEntry: "status"
[2020-10-07T12:16:24.782Z] Found binary from desktop file: "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/usr/bin/nim_status_client"
[2020-10-07T12:16:24.782Z] FHS-like mode with PREFIX, fhsPrefix: "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/usr"
[2020-10-07T12:16:24.782Z] app-binary: "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/usr/bin/nim_status_client"
[2020-10-07T12:16:24.782Z] appDirPath: "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist"
[2020-10-07T12:16:24.782Z] relativeBinPath: "usr/bin/nim_status_client"
[2020-10-07T12:16:24.782Z] Found icons from desktop file: ("/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/usr/status.svg", "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/status.svg")
[2020-10-07T12:16:24.782Z] preExistingToplevelIcon: "/home/jenkins/workspace/nim-status-client_linux_PR-1128/tmp/linux/dist/status.svg"
[2020-10-07T12:17:32.489Z] ERROR: ldd outputLine: "libsmime3.so => not found"
[2020-10-07T12:17:32.489Z] ERROR: for binary: "/opt/qt/5.14.0/gcc_64/qml/QtWebEngine/libqtwebengineplugin.so"
[2020-10-07T12:17:32.489Z] ERROR: Please ensure that all libraries can be found by ldd. Aborting.
[2020-10-07T12:17:32.489Z] Makefile:242: recipe for target 'pkg/StatusIm-201007-121021-b4ca07-pr.AppImage' failed
[2020-10-07T12:17:32.490Z] make: *** [pkg/StatusIm-201007-121021-b4ca07-pr.AppImage] Error 1
script returned exit code 2

@jakubgs
Copy link
Member

jakubgs commented Oct 7, 2020

Rebuilding image with added libnss3.

jakubgs added a commit that referenced this issue Oct 7, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
@jakubgs
Copy link
Member

jakubgs commented Oct 7, 2020

Looks like there's more stuff it will need:

jenkins@81df820f3a48:/home/user$ ldd /opt/qt/5.14.0/gcc_64/qml/QtWebEngine/libqtwebengineplugin.so | grep "not found"
    libXcomposite.so.1 => not found
    libXtst.so.6 => not found

jakubgs added a commit that referenced this issue Oct 7, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
richard-ramos pushed a commit that referenced this issue Oct 7, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
jakubgs added a commit that referenced this issue Oct 7, 2020
We do this here to avoid having to fork the original image repo:
https://github.com/a12e/docker-qt

Resolves: #523

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants