Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build(bindings/java): Allow building on linux-aarch_64 #3527

Merged
merged 2 commits into from
Nov 8, 2023
Merged

build(bindings/java): Allow building on linux-aarch_64 #3527

merged 2 commits into from
Nov 8, 2023

Conversation

amunra
Copy link
Contributor

@amunra amunra commented Nov 8, 2023

Allow building the Java bindings on Linux ARM64.

Additionally allows the script to run with older Python versions that require strings in subprocess calls.

Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks a lot!

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2023

Unfortunately, we lack the CI resources to accommodate this change, but it's likely to function.

@Xuanwo Xuanwo merged commit 9546db7 into apache:main Nov 8, 2023
53 checks passed
@amunra
Copy link
Contributor Author

amunra commented Nov 8, 2023

This obviously brings in the whole question of actually cutting the binaries for this platform.
For now, I've been adding incubator-opendal as a git submodule and pulling it by adding it as a module in pom.xml.

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2023

This obviously brings in the whole question of actually cutting the binaries for this platform. For now, I've been adding incubator-opendal as a git submodule and pulling it by adding it as a module in pom.xml.

https://github.com/apache/incubator-opendal/blob/9546db7d9736b4fcfd42713731942c443fb54dc4/.github/workflows/release_java.yml#L67-L75

We are performing cross compile for macos aarch64 support, maybe we can do it for linux aarch64 too.

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2023

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

That's impressive! I wasn't aware that opendal-java had a user. Is your project open source, perhaps questdb? May I include you as a listed user in the README.md file?

@amunra
Copy link
Contributor Author

amunra commented Nov 8, 2023

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

That's impressive! I wasn't aware that opendal-java had a user. Is your project open source, perhaps questdb? May I include you as a listed user in the README.md file?

How we use OpenDAL:

  • We have a closed-source edition of QuestDB called "Enterprise".
  • This release has replication support which works by storing WALs in an object store.
  • Our original implementation supported S3 only, last week we swapped out our implementation from S3 to opendal in our Rust code (our replication feature is written in Rust because we want to avoid garbage collection) in the interest of supporting Azure Blob.
  • I now have an incomplete PR that swaps out our Java tests from using the AWS Java SDK to using opendal-java. This is where the set of bugfixes are coming from.
  • Our DB builds on a number of platforms, including Linux/ARM64 which is where this patch comes in.
  • It would be awesome to get Linux/ARM64 binaries for opendal-java.
  • We intend to support additional object stores over time (webhdfs is probably next up), but this will be on a by-need basis dependent on what our customers are after.

Thanks for building this lib! It's coming together nicely!

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2023

Bravo!

@tisonkun
Copy link
Member

tisonkun commented Nov 8, 2023

maybe we can do it for linux aarch64 too.

The last time I try it, cargo build returns error. So at least some code doesn't support cross compile to aarch64 on x86_64 on linux.

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2023

The last time I try it, cargo build returns error. So at least some code doesn't support cross compile to aarch64 on x86_64.

Got it, let's track this in a new issue instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants