Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightweigth win32 installations #4100

Closed
totaam opened this issue Jan 18, 2024 · 9 comments
Closed

lightweigth win32 installations #4100

totaam opened this issue Jan 18, 2024 · 9 comments
Labels
enhancement New feature or request packaging win32 MS Windows OS

Comments

@totaam
Copy link
Collaborator

totaam commented Jan 18, 2024

Rather than having regular and "client" builds, with very little difference between them in terms of disk space:

$ du -sm \
    dists/windows/Xpra-Client-x86_64_Setup_5.0.4-r0.exe \
    dists/windows/Xpra-x86_64_Setup_5.0.4-r0.exe
123	dists/windows/Xpra-Client-x86_64_Setup_5.0.4-r0.exe
138	dists/windows/Xpra-x86_64_Setup_5.0.4-r0.exe

(the disk space actually used when installed is closer to 2GB, which is just ridiculous)

Split them into regular and "lightweigth" builds and remove more than just the server components: putty, remove all cuda and nvidia modules, all the video encoders, numpy (as long as pyopengl still works), ffmpeg codecs used by gstreamer, etc..
Use this as an exercise to also remove libraries we don't really need: clang / llvm?, libicu?, openblas, opencv, etc

This will also help with #4050

@totaam totaam added enhancement New feature or request win32 MS Windows OS packaging labels Jan 18, 2024
totaam added a commit that referenced this issue Mar 2, 2024
@totaam
Copy link
Collaborator Author

totaam commented Mar 2, 2024

With this initial commit, the size of the "Minimal" builds are now just:

  • installation directory: 181MB
  • EXE: 49MB
  • ZIP: 71MB

Because of the quickly diminishing returns, I think the bulk of the savings are already implemented.

Still TODO:

  • rename to "Light" builds?
  • gstreamer: remove video encoder, add build toggle for it?
  • remove enc_rgb?
  • start dialog missing
  • update the upload and release scripts, wiki, etc..
  • silence some warnings resulting from this trimming process - flag it?

totaam added a commit that referenced this issue Mar 3, 2024
* trim pixbuf loaders
* gstreamer DLLs
* remove python-lz4
* py.typed metadata
* header files
* backup, patch rejects
* don't bundle psutil in light builds
* move more python modules to the ZIP file
* add optimize for size switch: Os
* don't bundle pointless pkcs11 etc dir
* only ship the xpra config files actually used (no proxy or server in light builds)
* don't ship http headers or command metadata
totaam added a commit that referenced this issue Mar 4, 2024
* split argb modules from the encoder
* fix python modules zip multi-line command
* don't duplicate top-level DLLs
totaam added a commit that referenced this issue Mar 4, 2024
totaam added a commit that referenced this issue Mar 4, 2024
totaam added a commit that referenced this issue Mar 5, 2024
totaam added a commit that referenced this issue Mar 5, 2024
also remove more build / unused packaging files
@totaam
Copy link
Collaborator Author

totaam commented Mar 5, 2024

This will have to do for v6, though some work on this is still needed:

  • some form of backports for the egregious unnecessary weight
  • similar trimming for the macos builds
  • improvements to the SBOM records

Notes:

  • no nvidia decoders, no cuda, no numpy
  • no avif
  • no gstreamer video decoders
  • no encoders at all
  • no webcam
  • metadata removed
  • some icons / fonts removed
    etc

Latest builds:

$ du -sm beta/windows/*r35267*
69	beta/windows/Xpra-Light-x86_64_6.0-r35267.zip
49	beta/windows/Xpra-Light-x86_64_Setup_6.0-r35267.exe
175	beta/windows/Xpra-x86_64_6.0-r35267.zip
126	beta/windows/Xpra-x86_64_Setup_6.0-r35267.exe

And the installed uncompressed size:

171	Xpra-Light-x86_64_6.0-r35267
383	Xpra-x86_64_6.0-r35267

@stdedos you may be interested in using these builds if you don't have any nvidia GPUs in your clients.

@totaam
Copy link
Collaborator Author

totaam commented Mar 5, 2024

Comparing with v5.x at 1d716ad:

$ du -sm beta/windows/*5.0.8-r9.{exe,zip}
126	beta/windows/Xpra-Client-x86_64_Setup_5.0.8-r9.exe
141	beta/windows/Xpra-x86_64_Setup_5.0.8-r9.exe
177	beta/windows/Xpra-Client-x86_64_5.0.8-r9.zip
199	beta/windows/Xpra-x86_64_5.0.8-r9.zip

totaam added a commit that referenced this issue Mar 5, 2024
simplified version of the changes made in git master,
without the 'xpra gui' changes or '-Os'
totaam added a commit that referenced this issue Mar 5, 2024
as previously noted, nacl can't find its bindings when bundled in the library.zip file
@totaam
Copy link
Collaborator Author

totaam commented Mar 6, 2024

v5 is done and is almost as good as v6!

$ du -sm beta/windows/Xpra*5.0.8-r10*.{zip,exe}
78	beta/windows/Xpra-Light-x86_64_5.0.8-r10.zip
56	beta/windows/Xpra-Light-x86_64_Setup_5.0.8-r10.exe

However, v3.x is still absolutely massive:

$ du -sm Xpra*3.1.7-r77M*.{zip,exe}
349	/home/antoine/Shared/Xpra-Client-Python3-x86_64_3.1.7-r77M.zip
224	/home/antoine/Shared/Xpra-Client-Python3-x86_64_Setup_3.1.7-r77M.exe

And the installed size is ridiculous (almost 1TB!) - mostly because of opencv (#3336):

$ du -sm Xpra-Client-Python3-x86_64_3.1.7-r77M
973	Xpra-Client-Python3-x86_64_3.1.7-r77M

totaam added a commit that referenced this issue Mar 6, 2024
totaam added a commit that referenced this issue Mar 6, 2024
and some cosmetic spaces vs tabs
@totaam
Copy link
Collaborator Author

totaam commented Mar 6, 2024

The big difficulty with 3.x is that we have to ship ffmpeg to be able to decode h264 (the dec_avcodec2 decoder), whereas v5.x has openh264.
And ffmpeg brings in tons of dependencies, big ones too, so the resulting installation was still over 800MB!
So much so that we used to build our own ffmpeg package with trimmed down dependencies (#3166), just like we still do on MacOS.
But this was just too much effort to manage properly.

So I've decided to also remove ffmpeg from the "light" builds in v3.x - vp8 / vp9 are decent alternatives anyway, and truly free too. One could consider backporting openh264 to this branch, as it is self-contained.
Anyone really needing h264 can use the fat builds, or even better: a newer version.

There are still a few more stray DLLs in this version than the newer branches, but the "Light" build size is now much more reasonable:

$ du -sm beta/windows/Xpra*r84*.{zip,exe}
107	beta/windows/Xpra-Light-Python3-x86_64_3.1.7-r84.zip
73	beta/windows/Xpra-Light-Python3-x86_64_Setup_3.1.7-r84.exe

Top level:
image

lib directory:
image

@totaam
Copy link
Collaborator Author

totaam commented Mar 13, 2024

And within 24 hours of closing this ticket, an MSYS2 update brings in GStreamer version 1.24.0, which causes some new breakage - with a dialog alert box shown for every missing DLL!:

(Xpra_Audio.exe:1052): GStreamer-WARNING **: 20:48:51.306: \
    Failed to load plugin 'C:\Program Files\Xpra\lib\gstreamer-1.0\libgstd3d11.dll': The specified procedure could not be found.
(Xpra_Audio.exe:1052): GStreamer-WARNING **: 20:48:51.837: \
    Failed to load plugin 'C:\Program Files\Xpra\lib\gstreamer-1.0\libgstisomp4.dll': The specified procedure could not be found.
(Xpra_Audio.exe:1052): GStreamer-WARNING **: 20:48:53.017: \
    Failed to load plugin 'C:\Program Files\Xpra\lib\gstreamer-1.0\libgstd3d11.dll': The specified procedure could not be found.
(Xpra_Audio.exe:1052): GStreamer-WARNING **: 20:48:53.884: \
    Failed to load plugin 'C:\Program Files\Xpra\lib\gstreamer-1.0\libgstisomp4.dll': The specified procedure could not be found.

And even after looking at these DLLs using the dependency tracker, I have no idea what the problem is.

@totaam totaam reopened this Mar 13, 2024
@totaam
Copy link
Collaborator Author

totaam commented Mar 19, 2024

Though not as dramatic as v6, the 3.1.7 release got some significant weight loss:

du -sm Xpra*3.1.7*
73	Xpra-Light-Python3-x86_64_3.1.7-r0.msi
107	Xpra-Light-Python3-x86_64_3.1.7-r0.zip
72	Xpra-Light-Python3-x86_64_Setup_3.1.7-r0.exe
209	Xpra-Python3-x86_64_3.1.7-r0.msi
307	Xpra-Python3-x86_64_3.1.7-r0.zip
208	Xpra-Python3-x86_64_Setup_3.1.7-r0.exe

@stdedos
Copy link
Collaborator

stdedos commented Jun 26, 2024

@stdedos you may be interested in using these builds if you don't have any nvidia GPUs in your clients.

I do, but what's the benefit of "supporting" nvidia? CPU offload?

@totaam
Copy link
Collaborator Author

totaam commented Jun 27, 2024

I do, but what's the benefit of "supporting" nvidia? CPU offload?

Very fast GPU encoding and decoding.
For a client, that means accelerated jpeg decoding via nvdec or nvjpeg.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request packaging win32 MS Windows OS
Projects
None yet
Development

No branches or pull requests

2 participants