Worker: Add `make update-wrap-file` task #938

ibc · 2022-11-03T15:43:40Z

Usage example: make update-wrap-file SUBPROJECT=openssl.

Bonus Tracks

Make comments in Makefile respect max length ~80 columns.
Order tasks in Makefile.
Update libuv by using the new make update-subproject task.

- Usage example: `make update-subproject SUBPROJECT=openssl`. ### Bonus Tracks - Make comments in `Makefile` respect max length ~80 columns. - Order tasks in `Makefile`. - Update `libuv` by using the new `make update-subproject` task.

ibc · 2022-11-03T15:45:05Z

Off topic: I did this to update OpenSSL given the recent CVE-2022-3786 and CVE-2022-3602, however last version of OpenSSL in https://mesonbuild.com/Wrapdb-projects.html is still 3.0.2-1, which is really outdated :( How are we supposed to deal with this?

ibc · 2022-11-03T15:47:48Z

Ohhh.... why this? I didn't do anything in this PR that may have caused this...

https://github.com/versatica/mediasoup/actions/runs/3387127006/jobs/5627440846#step:6:124

jmillan · 2022-11-03T15:49:26Z

Ohhh.... why this? I didn't do anything in this PR that may have caused this...

Just do the changes suggested in the error. Can be new version of Meson requires it.

ibc · 2022-11-03T15:50:18Z

Ohhh.... why this? I didn't do anything in this PR that may have caused this...

Just do the changes suggested in the error. Can be new version of Meson requires it.

How can this PR cause that? I didn't change Meson version.

ibc · 2022-11-03T15:51:45Z

How can this PR cause that? I didn't change Meson version.

Change done.

jmillan · 2022-11-03T15:59:04Z

How can this PR cause that? I didn't change Meson version.

I believe is a meson auto update, or a fresh installation in github actions.

ibc · 2022-11-03T16:08:00Z

I'm updating openssl.wrap manually to force version 3.0.7 according to https://www.openssl.org/source/ (SHA256 included):

[wrap-file]
directory = openssl-3.0.7
source_url = https://www.openssl.org/source/openssl-3.0.7.tar.gz
source_filename = openssl-3.0.7.tar.gz
source_hash = 83049d042a260e696f62406ac5c08bf706fd84383f945cf21bd61e9ed95c396e

[provide]
libcrypto = libcrypto_dep
libssl = libssl_dep
openssl = openssl_dep

Then I run make clean-all and make clean-subprojects (just in case) and then make. Unfortunately it fails:

$ make
/Users/ibc/src/v3-mediasoup/worker/out/pip/bin/meson setup \
		--prefix /Users/ibc/src/v3-mediasoup/worker/out/Release \
		--bindir '' \
		--libdir '' \
		--buildtype release \
		-Db_ndebug=true \
		-Db_pie=true \
		-Db_staticpic=true \
		--reconfigure \
		"" \
		/Users/ibc/src/v3-mediasoup/worker/out/Release/build || \
		/Users/ibc/src/v3-mediasoup/worker/out/pip/bin/meson setup \
			--prefix /Users/ibc/src/v3-mediasoup/worker/out/Release \
			--bindir '' \
			--libdir '' \
			--buildtype release \
			-Db_ndebug=true \
			-Db_pie=true \
			-Db_staticpic=true \
			"" \
			/Users/ibc/src/v3-mediasoup/worker/out/Release/build
Directory does not contain a valid build tree:
/Users/ibc/src/v3-mediasoup/worker/out/Release/build
The Meson build system
Version: 0.61.5
Source dir: /Users/ibc/src/v3-mediasoup/worker
Build dir: /Users/ibc/src/v3-mediasoup/worker/out/Release/build
Build type: native build
Project name: mediasoup-worker
Project version: undefined
C compiler for the host machine: cc (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.102)")
C linker for the host machine: cc ld64 819.6
C++ compiler for the host machine: c++ (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.102)")
C++ linker for the host machine: c++ ld64 819.6
Host machine cpu family: x86_64
Host machine cpu: x86_64
Downloading openssl source from https://www.openssl.org/source/openssl-3.0.7.tar.gz
Download size: 15107575
Downloading: ..........

meson.build:163:0: ERROR: Subproject exists but has no meson.build file

ibc · 2022-11-03T16:10:33Z

I'm updating openssl.wrap manually to force version 3.0.7 according to https://www.openssl.org/source/ (SHA256 included):

[wrap-file]
directory = openssl-3.0.7
source_url = https://www.openssl.org/source/openssl-3.0.7.tar.gz
source_filename = openssl-3.0.7.tar.gz
source_hash = 83049d042a260e696f62406ac5c08bf706fd84383f945cf21bd61e9ed95c396e

[provide]
libcrypto = libcrypto_dep
libssl = libssl_dep
openssl = openssl_dep

Then I run make clean-all and make clean-subprojects (just in case) and then make. Unfortunately it fails:

$ make
/Users/ibc/src/v3-mediasoup/worker/out/pip/bin/meson setup \
		--prefix /Users/ibc/src/v3-mediasoup/worker/out/Release \
		--bindir '' \
		--libdir '' \
		--buildtype release \
		-Db_ndebug=true \
		-Db_pie=true \
		-Db_staticpic=true \
		--reconfigure \
		"" \
		/Users/ibc/src/v3-mediasoup/worker/out/Release/build || \
		/Users/ibc/src/v3-mediasoup/worker/out/pip/bin/meson setup \
			--prefix /Users/ibc/src/v3-mediasoup/worker/out/Release \
			--bindir '' \
			--libdir '' \
			--buildtype release \
			-Db_ndebug=true \
			-Db_pie=true \
			-Db_staticpic=true \
			"" \
			/Users/ibc/src/v3-mediasoup/worker/out/Release/build
Directory does not contain a valid build tree:
/Users/ibc/src/v3-mediasoup/worker/out/Release/build
The Meson build system
Version: 0.61.5
Source dir: /Users/ibc/src/v3-mediasoup/worker
Build dir: /Users/ibc/src/v3-mediasoup/worker/out/Release/build
Build type: native build
Project name: mediasoup-worker
Project version: undefined
C compiler for the host machine: cc (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.102)")
C linker for the host machine: cc ld64 819.6
C++ compiler for the host machine: c++ (clang 14.0.0 "Apple clang version 14.0.0 (clang-1400.0.29.102)")
C++ linker for the host machine: c++ ld64 819.6
Host machine cpu family: x86_64
Host machine cpu: x86_64
Downloading openssl source from https://www.openssl.org/source/openssl-3.0.7.tar.gz
Download size: 15107575
Downloading: ..........

meson.build:163:0: ERROR: Subproject exists but has no meson.build file

which makes sense since, indeed, there is no meson.build file in https://www.openssl.org/source/openssl-3.0.7.tar.gz BUT NEITHER in the previous https://www.openssl.org/source/openssl-3.0.2.tar.gz file!!!

ibc · 2022-11-03T16:21:56Z

So, so the thing is that, in previous openssl.wrap file there were patch fields:

[wrap-file]
directory = openssl-3.0.2
source_url = https://www.openssl.org/source/openssl-3.0.2.tar.gz
source_filename = openssl-3.0.2.tar.gz
source_hash = 98e91ccead4d4756ae3c9cde5e09191a8e586d9f4d50838e7ec09d6411dfdb63
patch_filename = openssl_3.0.2-1_patch.zip
patch_url = https://wrapdb.mesonbuild.com/v2/openssl_3.0.2-1/get_patch
patch_hash = 762ab4ea94d02178d6a1d3eb63409c2c4d61315d358391cdac62df15211174d4

[provide]
libcrypto = libcrypto_dep
libssl = libssl_dep
openssl = openssl_dep

This makes Meson fetch a special patch file from https://wrapdb.mesonbuild.com/v2/openssl_3.0.2-1/get_patch which is a zip file that contains meson.build and many other Meson related stuff :(

ibc · 2022-11-03T16:23:23Z

ok, this is too much for me...

Maybe caused by issue rust-lang/rust-clippy#9538 which may be fixed by PR rust-lang/rust-clippy#9662.

ibc · 2022-11-03T16:36:37Z

Ok, I've compared openssl-3.0.2 and openssl-3.0.7 taking into account the patched file of Meson for openssl-3.0.2. AFAIS nothing prevents us from using the same openssl_3.0.2-1_patch.zip from wrapdb.mesonbuild.com in openssl-3.0.7.

However it's not gonna work since the path zip must contain a folder with same name as the desired subproject to patch, which in our case is openssl-3.0.7, however openssl_3.0.2-1_patch.zip from wrapdb.mesonbuild.com contains a openssl-3.0.2 folder as content (as expected). So same meson.build not found error...

jmillan · 2022-11-03T16:49:56Z

However it's not gonna work since the path zip must contain a folder with same name as the desired subproject to patch,...

It's not that straightforward,

There are specific instructions to create/update meson wrap subprojects here https://mesonbuild.com/Adding-new-projects-to-wrapdb.html

ibc · 2022-11-03T16:59:33Z

There are specific instructions to create/update meson wrap subprojects here https://mesonbuild.com/Adding-new-projects-to-wrapdb.html

I see. However take a look to how complex making this for a new openss version is. This is the README in the patch file in mesonbuild for openssl-3.0.2:

OpenSSL for Meson

How this works?

TL;DR: this wrap abuses OpenSSL build system within Node.js.

Node.js has OpenSSL built-in with additional scripting around it to generate configs for GYP build system and thus bypass OpenSSL's native build system.

This wrap abuses that feature by replacing bundled OpenSSL with upstream version, patching mentioned mechanism to also generate a bunch of meson.build files for different platforms and uses top-level meson.build to wire everything together.

During installation unmodified Node.js tarball will be downloaded, its bundled OpenSSL will be replaced with upstream version and patched with meson.build files, enabling ability to build OpenSSL with Meson 🎉.

How to update to newer release

Unless Node.js changes the mechanism we abuse above (unlikely, but possible, please check the diff between corresponding versions), generator.sh file can be used.

Just update OpenSSL version in wrap file, update Node.js version in generator.sh file to such that contains matching OpenSSL version bundled with it and run generator.sh from the root of the repository:

subprojects/packagefiles/openssl/generator.sh

Generated files in generated-config directory, after which you can try to build it. create_release.py will run it as part of the release process, so it doesn't need to be included in Git.

Acknowledgement

This OpenSSL port wouldn't be possible without Node.js project under MIT license, whose OpenSSL build system was decomposed and heavily refactored.

jmillan · 2022-11-03T17:02:27Z

Indeed @nazar-pc did the previous one https://github.com/mesonbuild/wrapdb/commits/master/subprojects/openssl.wrap

ibc · 2022-11-03T17:02:54Z

I'll create a new ticket for the openssl update stuff and leave this PR just for what it was intended: add a new make task update-wrap-file.

nazar-pc · 2022-11-03T17:03:54Z

How can this PR cause that? I didn't change Meson version.

This and other might be related to Rust version upgrade. Run rustup update and it may fix those if you were running an older version. If you're running a newer instead you (or I) will need to fix those, clippy improves suggestions over time, adds new lints and improves existing ones. We have CI configured to not allow any warnings whatsoever.

As to OpenSSL update you'll need to update it in Meson wraps first unless someone else did it. According to https://mesonbuild.com/Wrapdb-projects.html version there is quite old and needs to be updated, you can see one of the PRs I did and follow readme as you already noticed. For instance here: mesonbuild/wrapdb#281

Same about libuv, here is an example of upgrade (basically read upstream CMakeLists changes and tweak meson.build): mesonbuild/wrapdb#367

nazar-pc · 2022-11-03T17:07:14Z

New Rust version was just released earlier today, so this is probably why you hit those warnings, I'll send a PR fixing them.

ibc · 2022-11-03T17:16:22Z

Guys, let's please manage the OpenSSL version issue in here: #939

@nazar-pc assuming that you are gonna fix the Rust "clippy" related issues in a separate PR, are we good with this PR?

nazar-pc

@nazar-pc assuming that you are gonna fix the Rust "clippy" related issues in a separate PR, are we good with this PR?

Yes, looks fine to me

Worker: Add make update-subproject task

34c0518

- Usage example: `make update-subproject SUBPROJECT=openssl`. ### Bonus Tracks - Make comments in `Makefile` respect max length ~80 columns. - Order tasks in `Makefile`. - Update `libuv` by using the new `make update-subproject` task.

ibc requested review from jmillan and nazar-pc November 3, 2022 15:43

jmillan approved these changes Nov 3, 2022

View reviewed changes

Attempt to fix CI error in worker/build.rs

446a131

Rename update-subproject to update-wrap-file

082cfcb

ibc changed the title ~~Worker: Add make update-subproject task~~ Worker: Add make update-wrap-file task Nov 3, 2022

fix doc/Building.md

0cfa01b

nazar-pc approved these changes Nov 3, 2022

View reviewed changes

ibc merged commit e3cf83d into v3 Nov 3, 2022

ibc deleted the add-make-update-subproject branch November 3, 2022 18:43

piranna pushed a commit to dyte-in/mediasoup that referenced this pull request Feb 9, 2023

Worker: Add make update-wrap-file task (versatica#938)

fb5678c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Worker: Add `make update-wrap-file` task #938

Worker: Add `make update-wrap-file` task #938

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

nazar-pc commented Nov 3, 2022

nazar-pc commented Nov 3, 2022

ibc commented Nov 3, 2022

nazar-pc left a comment

Worker: Add make update-wrap-file task #938

Worker: Add make update-wrap-file task #938

Conversation

ibc commented Nov 3, 2022 • edited

Bonus Tracks

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

ibc commented Nov 3, 2022 • edited

ibc commented Nov 3, 2022 • edited

ibc commented Nov 3, 2022 • edited

ibc commented Nov 3, 2022 • edited

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

OpenSSL for Meson

How this works?

How to update to newer release

Acknowledgement

jmillan commented Nov 3, 2022

ibc commented Nov 3, 2022

nazar-pc commented Nov 3, 2022

nazar-pc commented Nov 3, 2022

ibc commented Nov 3, 2022

nazar-pc left a comment

Choose a reason for hiding this comment

Worker: Add `make update-wrap-file` task #938

Worker: Add `make update-wrap-file` task #938

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited

ibc commented Nov 3, 2022 •

edited