-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Meson build system 💪 #622
Meson build system 💪 #622
Conversation
788ec54
to
f2998ac
Compare
This is awesome. Please let me some days to review because I'm from here to there. |
For what it's worth, I use this on Windows and really appreciate the effort there. I am curious if there are performance penalties on Windows with no-ASM OpenSSL. I am hoping with this build system it'll be easy to have the Rust lib support the MSVC target too. |
GYP version also used non-ASM version of OpenSSL, so nothing changes about that yet. On Rust side file descriptor communication is the primary and basically only thing that is not implemented (I don't use Windows, so unless someone contributes it is not gonna happen), but it is out of scope of this PR, please create a topic on the forum if you want to work on that and I'll try to help you with that. |
5432aa7
to
2abffbc
Compare
I don't see the point of bundling a 25k lines copy of get-pip.py, can't you just run |
@eli-schwartz it would have been easier indeed, but if you do that on Ubuntu 20.04 for instance, you'll get this:
That is why I decided to bundle |
You need to install python3-venv on Debian (or, like the rest of the world, stop using Debian-built python for anything -- theirs is the only distro where the package is actively problematic to use by users instead of distro developers). Alternatively you could IMHO anything is better than checking 25k lines into git. |
Requiring Cloning Meson adds tools other than Python3 (like I'm fine either way (we already bundle the whole OpenSSL here, which far bigger), if Inaki and Jose think adding PIP to requirements is fine, I'll update PR accordingly. |
https://packages.ubuntu.com/focal/amd64/python3.8-venv/filelist This package has the ensurepip module in it. python3-venv depends on python3-{minor_version}-venv. ... Personally I do not and never have seen the problem with installing packages from a distro... Ubuntu 20.04 has meson 0.53 and ninja 1.10 so if you |
The challenge here is that mediasoup users interact with NPM and Cargo. They don't care too much about how underlying modules are build and will be surprised when after minor update it suddenly breaks somewhere internally. And I'm not against what you're suggesting BTW, just trying to explain the rationale here. |
Yes, but on windows you generally don't need to install pip since it's installed by default. Anyway, I get the rationale, and am merely cautioning that it could be done without a specific disadvantage. For CI, installing python3-venv or python3-pip should be enough. For end users, I think running |
Finally last fork of mine was upstreamed, this PR relies on upstream releases/git revisions/wraps. |
Oh great!! BTW Sorry for the delay, Jose is off this week |
@nazar-pc please take a look to libsrtp 2.4.0 latest version, it comes with Meson support! |
@ibc this PR used pre-release version of 2.4.0 all along, happy to see official release, will update accordingly now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
A bit offtopic: Would this be in a 4.0.0 version, or a major version change is not planned? If so, where can we see the upcoming changes? I have some to propose... |
I do not understand why Windows tests are failing, it doesn't make any sense to me:
Sometimes it succeeds, sometimes it crashes. Unless anyone cares enough we might have to remove it from CI until further investigation. |
In tests.cpp can you enable mediasoup worker ERROR logs? I think there is a env that can be set in command line (it should be obvious by taking a look to tests.cpp) |
Oh but actually all tests passed... |
Maybe it's just a wrong non zero return value so npm task assumes it went wrong? |
IDK, it sometimes passes CI, for instance commit "Faster worker CI by skipping rebuilding due to installation cleanup" finished successfully here: https://github.com/nazar-pc/mediasoup/runs/3836364875?check_suite_focus=true |
Running this branch locally in macOS BigSur Intel. Some things:
Is
Should we fill
Too many compilation warnings in libuv. Can we avoid them? |
Yes, that is because Debian/Ubuntu have special version of Python, so we try 2 versions of the command expecting at least one to succeed. In your case it is second one, so it should still work fine.
We might if you want to maintain yet another place for version change. I don't think it matters too much in this case.
It should have not been there, we already set |
This PR should remove |
It should remove most of |
BTW I'm creating a PR on top of yours right now. Will announce it in less than 1-2 hours if everything goes fine. |
Here we are: nazar-pc#2 |
Do you really want to merge that PR into this branch or maybe we can merge it separately? This PR is really big already and nazar-pc#2 is rather distinct in what it contains. |
If I make my PR target current v3 branch, then lot of conflicts would happen later in yours. I've tested my PR (everything I could) and there are no issues. If we wait and any So yes, my intention is to merge this into your PR so we an move forward easily. I promise no regressions will happen due to my PR 😅 |
Can we just merge this first, create another PR and merge that one? At least there will be 2 distinct merges happening and not one huge update with everything at once. BTW, do you want me to comment-out Windows CI job for now? |
As far as you promise that you are not gonna modify any .ts or .js file... ok :)
Yes, please. BTW I would also remove |
I will not.
I'm a huge fan of meaningful distinct PRs and commits, this is why I spent so much time before submitting this PR such that someone can click on each commit and get something that makes sense on its own and compiles at every step of the way. So you can use tools like |
So.... let's wait for CI and merge!!!! |
please merge! |
Congrats on the successful conversion!
Whew, and that's even without deleting deps that meson can acquire for you. :D |
Those dependencies are just 4,700,569 lines of code, no big deal 🙃 |
I'll add a reminder tomorrow to make a new mediasoup release, announce it in the forum and update the website. THANKS |
What are the steps to setup the compiler for windows?
|
This PR replaces GYP build system with fully-functional Meson build system
🤪 How to review this PR
First of all, please read this message start to finish.
Please review this PR commit by commit, I have intentionally separated changes into pieces such that they make sense, every commit compiles without errors on Linux/macOS (Windows support is added in later commits) and large changes (like moving OpenSSL from one directory to another) are isolated in separate trivial-to-review commits for your convenience.
😎 Compatibility
Make and NPM commands are left are exactly the same as before, so no one should notice any difference.
Makefile has a few internal tasks added and clean/clean-all have slightly different meaning, but behavior is in line with what it was before, no one should be hurt.
Previously Python 2 or 3 were required on Linux/macOS, now strictly Python 3 with PIP is required.
On Windows Python 2 was required, now Python 3 with PIP is required just like on Linux/macOS + there is a new requirement to have Make (from MinWG).
In both cases changes are relatively minor, but they are breaking changes nevertheless!
⚙️ Meson configs and dependencies
The way things are implemented is the following:
meson.build
file is written for mediasoup-specific targets and libwebrtc codemeson.build
is used in the meantimefor OpenSSL I actually wrote a custommeson.build
config generator based on OpenSSL maintenance documentation in Node.js project, it is similar, but slightly custom comparing to upstream wrap proposal I submitted in Add OpenSSL mesonbuild/wrapdb#113 (because we still use the same Node.js fork with QUIC protocol support and that PR uses upstream release without QUIC and doesn't support ASM in the first revision that we do)customized generator scripts are available in https://github.com/nazar-pc/wrapdb/tree/openssl-mediasoup in case OpenSSL upgrade is necessary before there is an upstream wraponce my upstream wrap is released, OpenSSL bundled with the project will be removed from mediasoup repositorylibsrtp2
used is from my fork with exactly one commit on top and will be switched to upstream once my PR Improve OpenSSL KDF check cisco/libsrtp#546 is mergedusrsctp
is used from my fork with exactly one commit on top and will be switched to upstream once my PR Change defines to type aliases for better compatibility sctplab/usrsctp#601 is mergedabseil-cpp
uses upstream release and customized wrap, will be switched to upstream wrap once my PR Fix Abseil atomic lib lookup on VS. mesonbuild/wrapdb#116 and Don't define min/max macros on MSVC, it will cause compile errors for abseil-cpp mesonbuild/wrapdb#118 are mergednetstring-c
is currently bundled, will switch to upstream wrap once my PR Add netstring-c mesonbuild/wrapdb#115 is mergedabseil-cpp
,libuv
,libsrtp2
,catch2
,netstring-c
,nlohmann_json
,openssl
andusrsctp
come from unmodified upstream wraps or reposNOTE: Even though multiple forks of mine are used, they are all behind Sha256 hashes, so can be audited and will never change (I do not plan to remove corresponding branches in the near future either, so installations shouldn't break)🪄 How it works
Makefile is modified to first install PIP (with bundled installer) into mediasoup-specific directory, then Meson and Ninja are installed using that PIP installation. This way we only need Python 3 installed and nothing else.
Xcode generator is replaced with Meson, just like
compile_commands_template.json
is also generated by Meson and doesn't need to be included and re-generated manually with GYP+Bear combo. I didn't test either, but I expect them to work just fine (except different location of Xcode project).After Meson and Ninja are installed, they are used to setup and build existing Makefile targets. There are environment variables to configure Meson arguments, so it is possible to impact optimization level, ASAN builds, LTO and other options from the outside.
🎉 CI and Windows
All existing CI targets still work, but on top of those Windows CI targets are added:
GYP_MSVS_VERSION
is no longer needed to be specified, Meson will take care of everything♾️ Next steps
first of all, I need to continue work with upstream to land required changes in various places so that we no longer depend on my forks and I don't have to maintain them (and PRs are already submitted, just waiting for them to be reviewed)we can install PIP even if it is not present on the system already, relaxing system requirements further, but that would require to bundle(decided it is not worth it)get-pip.py
scriptthere is a minor inconvenience that only maintainers will notice when doing changes tomeson.build
file or dependencies caused by Ninja doesn't forward environment variables tomeson
on regeneration ninja-build/ninja#1997can be worked around withmake clean-all
, but would be nice to get it fixed, thankfully as soon as the fix lands, we'll be able to use it since we do not depend on Ninja being installed on the systemP.S. I spent way too much time on this, especially fighting with Windows while waiting for CI to finish over and over again, hopefully it gets through 🤞
Fixes #359, closes #602