Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-4917][VL] static packaging refine #5033

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

zhouyuan
Copy link
Contributor

What changes were proposed in this pull request?

refine the static package in vcpkg

  • bump boost to 1.84
  • adding duckdb
  • fix xsimd/simdjson version

How was this patch tested?

pass GHA static build test

@zhouyuan zhouyuan changed the title [GLUTEN-4917] static packaging refine [GLUTEN-4917][VL] static packaging refine Mar 19, 2024
Copy link

#4917

@zhouyuan zhouyuan force-pushed the wip_static_packaging_refine branch 7 times, most recently from 9b59d82 to 7d95f2b Compare March 27, 2024 06:30
@ulysses-you
Copy link
Contributor

@zhouyuan Does this pr ready ?

@zhouyuan
Copy link
Contributor Author

@zhouyuan Does this pr ready ?

Hi @ulysses-you

This patch adds the arrow build in vcpkg. it's not full ready for your case yet. The gap is:

  • currently Velox will force build libarrow/libparquet, need to add one feature to allow velox to reuse the pre-built arrow
  • I also find the compile flags for arrow are different in velox vs. vcpkg, may need some adjustments

the first patch may need some discussion with velox community. I'll try to do some tests in my own branch first.

Thanks,
-yuan

@ulysses-you
Copy link
Contributor

thank you @zhouyuan !

@zhouyuan zhouyuan force-pushed the wip_static_packaging_refine branch 5 times, most recently from 4525398 to 9d4e893 Compare April 1, 2024 09:26
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
@zhouyuan zhouyuan force-pushed the wip_static_packaging_refine branch from f082dc3 to ba178c4 Compare April 1, 2024 12:00
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
@zhouyuan zhouyuan force-pushed the wip_static_packaging_refine branch from ba178c4 to e8aae1d Compare April 1, 2024 23:49
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
@zhouyuan
Copy link
Contributor Author

zhouyuan commented Apr 2, 2024

Velox is able to find the pre-built arrow with 6b0fb02
gluten-cpp and the library loader needs to be modified for next

@ulysses-you
Copy link
Contributor

hi @zhouyuan if there any blocker on this ?

@zhouyuan
Copy link
Contributor Author

zhouyuan commented May 8, 2024

hi @zhouyuan if there any blocker on this ?

@ulysses-you

hi Xiduo, we are actually breaking up this patch into smaller ones to get merged. still following on this.

The first half is merged already however we do find some unit tests failed due to the new compile method. (free(): invalid pointer:)
#5515
We should fix the unit tests firstly, @JkSelf is working on this.

For static building with arrow and parquet(the second half of this patch) it's working on local test. However it requires to modify the library loading code path - which is actually conflicting with the original dynamic packaging method(as we still need to load libarrow/libparquet there). We may need to keep the both code path for a while as static packaging is not verified in production env.

thanks
-yuan

Copy link

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Jun 23, 2024
@zhouyuan zhouyuan removed the stale stale label Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants