Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mesa] Features, dependencies and CI #36081

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

dg0yt
Copy link
Contributor

@dg0yt dg0yt commented Jan 9, 2024

Revises dependencies and configuration.
Add feature for controlling vulkan support.
Enable CI.

@dg0yt dg0yt force-pushed the mesa-vulkan branch 2 times, most recently from 330b098 to 6ad59c4 Compare January 9, 2024 07:49
@Cheney-W Cheney-W added category:new-port The issue is requesting a new library to be added; consider making a PR! category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist labels Jan 9, 2024
@dg0yt dg0yt changed the title [mesa] WIP [mesa] Dependencies and CI updates Jan 13, 2024
@@ -74,7 +59,7 @@
]
},
"llvm": {
"description": "Build with llvmpipe",
"description": "Build with LLVM",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"description": "Build with LLVM",
"description": "Build llvmpipe",

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feature doesn't only control llvmpipe but also some other drivers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't most (all) of them just depend on llvmpipe and not llvm itself?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the feature would be called "llvmpipe", I probably wouldn't have touched it like this.

just depend on llvmpipe and not llvm itself

At least mesa also wants llvm[target-amdgpu] for with_gallium_radeonsi.
So either llvm becomes a mandatory dependency (rendering feature llvm nearly pointless), or the feature really controls the llvm dependency (this PR).

@dg0yt dg0yt marked this pull request as ready for review January 15, 2024 05:07
Cheney-W
Cheney-W previously approved these changes Jan 15, 2024
@Cheney-W Cheney-W added the info:reviewed Pull Request changes follow basic guidelines label Jan 15, 2024
@JavierMatosD JavierMatosD added the depends:vm-update PR contains changes to the VM provisioning scripts label Jan 17, 2024
JavierMatosD
JavierMatosD previously approved these changes Jan 17, 2024
Copy link
Contributor

@JavierMatosD JavierMatosD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm putting this PR on hold for now. We need to update our VMs to include libxcb-dri3-dev in our Linux CI environment, as the changes to scripts/azure-pipelines/linux/managed-image.json don't actually change anything. Tagging this as depends:vm-update. Once our VMs are updated, someone can pick this up and have all the context they need. Here's a quick rundown of what's in the PR:

  1. Shared GLAPI Condition: Now builds shared GLAPI if either OpenGL API or EGL is enabled. Previously, it was limited to OpenGL API.
  2. Valgrind Dependency: Removed the dependency on Valgrind.
  3. VM Configuration: Switched -Dshared-llvm from auto to disabled. The auto-detection was failing because LLVM is ONLY_STATIC_LIBRARY.
  4. gslangValidator: Added to ADDITIONAL_BINARIES if glslang is available.
  5. CI Environment: Updated to add libxcb-dri3-dev, but this depends on our VM update.
  6. In vcpkg.json: Simplified the default features list, introduced optional Vulkan support, and expanded LLVM support, including for AMD GPUs.

I'll revisit this PR once our VMs are all set. Until then, it's on hold but ready for action post-update. 🚀

@dg0yt, feel free to jump in if I've missed something or if there's anything that needs correction.

@JavierMatosD JavierMatosD removed the info:reviewed Pull Request changes follow basic guidelines label Jan 24, 2024
BillyONeal added a commit to BillyONeal/vcpkg that referenced this pull request Feb 14, 2024
BillyONeal added a commit that referenced this pull request Feb 22, 2024
* Cherry-pick infrastructure changes from
#36081 as authored by @dg0yt
* Suppress warnings for the monthly docker mint as suggested by Toddy
Mladenov ( MS internal conversation
https://teams.microsoft.com/l/message/19:f498f444e97f40b2a2027d015a9047ab@thread.tacv2/1707520630587?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=2080ccbe-0cfb-49e6-a9c1-ea6a6bf82722&parentMessageId=1707520630587&teamName=Containers%20Secure%20Supply%20Chain&channelName=3P%20Registry%20and%20Images%20Guidance&createdTime=1707520630587
)
* Move Azure Container Registry for Android to WestUS3; with this we are
officially entirely within one Azure region again :D
* Explicitly grant Azure DevOps permissions to touch the scale set
rather than relying on subscription-level assignments.
@BillyONeal BillyONeal removed the depends:vm-update PR contains changes to the VM provisioning scripts label Feb 22, 2024
@Cheney-W
Copy link
Contributor

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@dg0yt
Copy link
Contributor Author

dg0yt commented Feb 22, 2024

@BillyONeal et al. Artifact uploads seems to broken, error: curl failed to put file to ... everywhere, https://dev.azure.com/vcpkg/public/_build/results?buildId=99766&view=logs&j=7922e5c4-0103-5f8f-ad17-45ce9bb98e80.

TomKatom pushed a commit to TomKatom/vcpkg that referenced this pull request Feb 23, 2024
* Cherry-pick infrastructure changes from
microsoft#36081 as authored by @dg0yt
* Suppress warnings for the monthly docker mint as suggested by Toddy
Mladenov ( MS internal conversation
https://teams.microsoft.com/l/message/19:f498f444e97f40b2a2027d015a9047ab@thread.tacv2/1707520630587?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=2080ccbe-0cfb-49e6-a9c1-ea6a6bf82722&parentMessageId=1707520630587&teamName=Containers%20Secure%20Supply%20Chain&channelName=3P%20Registry%20and%20Images%20Guidance&createdTime=1707520630587
)
* Move Azure Container Registry for Android to WestUS3; with this we are
officially entirely within one Azure region again :D
* Explicitly grant Azure DevOps permissions to touch the scale set
rather than relying on subscription-level assignments.
Osyotr pushed a commit to Osyotr/vcpkg that referenced this pull request Feb 26, 2024
* Cherry-pick infrastructure changes from
microsoft#36081 as authored by @dg0yt
* Suppress warnings for the monthly docker mint as suggested by Toddy
Mladenov ( MS internal conversation
https://teams.microsoft.com/l/message/19:f498f444e97f40b2a2027d015a9047ab@thread.tacv2/1707520630587?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=2080ccbe-0cfb-49e6-a9c1-ea6a6bf82722&parentMessageId=1707520630587&teamName=Containers%20Secure%20Supply%20Chain&channelName=3P%20Registry%20and%20Images%20Guidance&createdTime=1707520630587
)
* Move Azure Container Registry for Android to WestUS3; with this we are
officially entirely within one Azure region again :D
* Explicitly grant Azure DevOps permissions to touch the scale set
rather than relying on subscription-level assignments.
@yurybura
Copy link
Contributor

yurybura commented Mar 27, 2024

@dg0yt Are there plans to complete these changes? I'm interested in updating Mesa to the latest build to support LLVM 18. See #37599.

@dg0yt
Copy link
Contributor Author

dg0yt commented Mar 27, 2024

Are there plans to complete these changes? I'm interested in updating Mesa to the latest build to support LLVM 18. See #37599.

@yurybura Yes. The delay is from getting additional system packages applied to the CI images, to enable CI for linux and osx. This happens once in a month... I would do a mesa update once CI works for these platforms.
That being said, I consider llvm significant enough to accept parallel mesa update work. It should perhaps be limited to minimal changes within the current CI coverage.

@dg0yt
Copy link
Contributor Author

dg0yt commented Apr 6, 2024

Any other meson port invoking CMake's check_c_source_compiles?
mesonbuild/meson#13049

@yurybura
Copy link
Contributor

@dg0yt Is there any progress in this PR?

@dg0yt
Copy link
Contributor Author

dg0yt commented Apr 23, 2024

Not so soon.
On x64-linux, there is now an LLVM issue:

/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libLLVMSupport.a(Process.cpp.o): in function `terminalHasColors(int)':
/mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:354: undefined reference to `set_curterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:356: undefined reference to `setupterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:374: undefined reference to `tigetnum'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:380: undefined reference to `set_curterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:381: undefined reference to `del_curterm'

I think I have been there before, some kind of installation order issue. But I can't rebuild LLVM so many time as I would need to.

@yurybura
Copy link
Contributor

I'm trying to support at least mesa:x64-windows. The meson tool needs to be updated and patched to support LLVM 18.1.x (see mesonbuild/meson@dec93ce).

@dg0yt
Copy link
Contributor Author

dg0yt commented Apr 23, 2024

As I said, go ahead with llvm as you like. Just keep changes to mesa minimal.

@yurybura yurybura mentioned this pull request May 17, 2024
7 tasks
BillyONeal pushed a commit that referenced this pull request Jun 10, 2024
Mitigation for #39198.

Mesa CI is currently disabled for linux and osx, and also doesn't run
for android.
So we see CI effects only for Windows. Improving CI situation for mesa
is the subject of #36081. So keeping egl out of CI like before
#37599 is the best option at the
moment.
@dg0yt dg0yt mentioned this pull request Jun 12, 2024
@dg0yt dg0yt changed the title [mesa] Dependencies and CI updates [mesa] Features, dependencies and CI Jun 12, 2024
@dg0yt
Copy link
Contributor Author

dg0yt commented Jun 12, 2024

Not so soon. On x64-linux, there is now an LLVM issue:

/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libLLVMSupport.a(Process.cpp.o): in function `terminalHasColors(int)':
/mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:354: undefined reference to `set_curterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:356: undefined reference to `setupterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:374: undefined reference to `tigetnum'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:380: undefined reference to `set_curterm'
/usr/bin/ld: /mnt/vcpkg-ci/b/llvm/src/org-17.0.2-5fdca9a361.clean/llvm/lib/Support/Unix/Process.inc:381: undefined reference to `del_curterm'

I think I have been there before, some kind of installation order issue. But I can't rebuild LLVM so many time as I would need to.

Welcome to meson's incomplete ability to import transitive dependencies from CMake, paired with vcpkg's incomplete ability to provide a consistent foo-config script experience.

@WangWeiLin-MV WangWeiLin-MV linked an issue Jul 11, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:new-port The issue is requesting a new library to be added; consider making a PR! category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[mesa] Build failure on arm64-osx
6 participants