Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port upstream NumPy 1.25.0 #1

Merged
merged 3 commits into from
Aug 17, 2023
Merged

Port upstream NumPy 1.25.0 #1

merged 3 commits into from
Aug 17, 2023

Conversation

andreittr
Copy link
Contributor

This change set ports the popular NumPy python extension to unikraft at upstream version 1.25.0, along with necessary compatibility patches.

This is part of a larger work and depends on Python 3.10

and all its dependencies; plus a patch to register numpy modules:

In addition:

Libs used for build:

LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-libcxx
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-libcxxabi
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-libunwind
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-intel-intrinsics
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-compiler-rt
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-musl
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-libffi
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-lwip
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-bzip2
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-sqlite
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-openssl
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-libuuid
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-zlib
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-python3
LIBS-y := $(LIBS-y):$(UK_LIBS)/lib-python-numpy

This was referenced Jul 25, 2023
Closed
Closed
Merged
@andreittr
Add README and CONTRIBUTING
d4b6296 
Signed-off-by: Andrei Tatar <andrei@unikraft.io>
@andreittr
Port upstream NumPy 1.25.0
0e5987a 
This work is based off NumPy 1.25.0, configured on a Linux x86_64
machine and adapted to work with unikraft. Hardware acceleration assumes
x86_64 and is dispatched at runtime. ARM support requires future work.

Files under `generated/` are created by the upstream build process from
templates; we include these files verbatim for the unikraft build.
Any potential patches should be clearly marked to ease future updates.
Files under `importfix/` are correctly namespaced wrapper modules for
any binary modules that we compile and link into unikraft, and must be
copied over to the python rootfs.
Selecting this library will add its files to the python rootfs build.

Signed-off-by: Andrei Tatar <andrei@unikraft.io>
@andreittr
Account for compiler support of CPU features
75a7382 
GCC < 12 and clang < 14 do not support certain x86 ISA extensions used
in runtime dispatch targets.
This change conditions compiling these targets on compiler version.

Signed-off-by: Andrei Tatar <andrei@unikraft.io>
@razvand razvand self-assigned this Aug 10, 2023
@razvand razvand added the enhancement New feature or request label Aug 10, 2023
StefanJum
StefanJum approved these changes Aug 10, 2023
View reviewed changes
Copy link
Member

@StefanJum StefanJum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good @andreittr, thank you.
I'll add the tag after the x86 instrinsics pr is merged.

RaduNichita
RaduNichita approved these changes Aug 11, 2023
View reviewed changes
Copy link

@RaduNichita RaduNichita left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good with me. I tested with gcc12 and clang15. Thanks @andreittr for this monstrous effort!

Reviewed-by: Radu Nichita radunichita99@gmail.com

@RaduNichita
Copy link

One small request, @andreittr: could you please add compiler version checks in the Makefile.uk (similar to what it is in the Unikraft Makefile).

You can see a similar discussion here.

@andreittr
Copy link
Contributor Author

One small request, @andreittr: could you please add compiler version checks in the Makefile.uk (similar to what it is in the Unikraft Makefile).

You can see a similar discussion here.

Do we have versions that are known to be too old to build numpy but newer than the ones in main Unikraft? Because I wouldn't want to error out just on a guess/the extent to which we've tested.

StefanJum
StefanJum approved these changes Aug 14, 2023
View reviewed changes
Copy link
Member

@StefanJum StefanJum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed-by: Stefan Jumarea stefanjumarea02@gmail.com

@andreittr
Copy link
Contributor Author

andreittr commented Aug 14, 2023 •
edited
Loading

Do we have versions that are known to be too old to build numpy but newer than the ones in main Unikraft? Because I wouldn't want to error out just on a guess/the extent to which we've tested.

Re: this, currently investigating an issue building with gcc 11, but it has to do with x86 intrinsics more broadly; so not really a numpy issue, will fix in lib-intel-intrinsics. Rootfs build is also ok. I think this can be merged.

edit: intrinsics fix in following PR:

razvand
razvand approved these changes Aug 14, 2023
View reviewed changes
Copy link
Contributor

@razvand razvand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved-by: Razvan Deaconescu razvand@unikraft.io

@razvand razvand merged commit 096d527 into unikraft:staging Aug 17, 2023
razvand pushed a commit that referenced this pull request Aug 17, 2023
@andreittr @razvand
Add README and CONTRIBUTING
4e63d39 
Signed-off-by: Andrei Tatar <andrei@unikraft.io>
Reviewed-by: Stefan Jumarea <stefanjumarea02@gmail.com>
Reviewed-by: Radu Nichita <radunichita99@gmail.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
GitHub-Closes: #1
razvand pushed a commit that referenced this pull request Aug 17, 2023
@andreittr @razvand
Port upstream NumPy 1.25.0
50ff1f9 
This work is based off NumPy 1.25.0, configured on a Linux x86_64
machine and adapted to work with unikraft. Hardware acceleration assumes
x86_64 and is dispatched at runtime. ARM support requires future work.

Files under `generated/` are created by the upstream build process from
templates; we include these files verbatim for the unikraft build.
Any potential patches should be clearly marked to ease future updates.
Files under `importfix/` are correctly namespaced wrapper modules for
any binary modules that we compile and link into unikraft, and must be
copied over to the python rootfs.
Selecting this library will add its files to the python rootfs build.

Signed-off-by: Andrei Tatar <andrei@unikraft.io>
Reviewed-by: Stefan Jumarea <stefanjumarea02@gmail.com>
Reviewed-by: Radu Nichita <radunichita99@gmail.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
GitHub-Closes: #1
razvand pushed a commit that referenced this pull request Aug 17, 2023
@andreittr @razvand
Account for compiler support of CPU features
3f4d917 
GCC < 12 and clang < 14 do not support certain x86 ISA extensions used
in runtime dispatch targets.
This change conditions compiling these targets on compiler version.

Signed-off-by: Andrei Tatar <andrei@unikraft.io>
Reviewed-by: Stefan Jumarea <stefanjumarea02@gmail.com>
Reviewed-by: Radu Nichita <radunichita99@gmail.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
GitHub-Closes: #1
@andreittr andreittr deleted the ttr/up-1.25.0 branch August 17, 2023 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@razvand razvand razvand approved these changes

@RaduNichita RaduNichita RaduNichita approved these changes

@StefanJum StefanJum StefanJum approved these changes

Assignees

@razvand razvand

Labels
enhancement New feature or request
Projects
(Binary) Application Compatibility
Status: Done
Unikraft Core Roadmap
Status: ✅ Done!
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@andreittr @RaduNichita @razvand @StefanJum