Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-7645: [Packaging][deb][RPM] Fix arm64 packaging build #6253

Closed
wants to merge 4 commits into from
Closed

ARROW-7645: [Packaging][deb][RPM] Fix arm64 packaging build #6253

wants to merge 4 commits into from

Conversation

kou
Copy link
Member

@kou kou commented Jan 22, 2020

qemu-user-static package is updated.

@kou
[Packaging][deb][RPM] Fix arm64 packaging build
7a52b5f 
qemu-user-static package is updated.
@kou
Copy link
Member Author

kou commented Jan 22, 2020 •
edited

@ursabot crossbow submit -g linux-arm

@kszucs
Copy link
Member

kszucs commented Jan 22, 2020

AMD64 Conda Crossbow Submit (#87603) builder has been succeeded.

Revision: 7a52b5f

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-445

Task Status
centos-7-aarch64 Azure
centos-8-aarch64 Azure
debian-buster-arm64 Azure
debian-stretch-arm64 Azure
ubuntu-bionic-arm64 Azure
ubuntu-disco-arm64 Azure
ubuntu-eoan-arm64 Azure
ubuntu-xenial-arm64 Azure

@github-actions
Copy link

https://issues.apache.org/jira/browse/ARROW-7645

kszucs
kszucs approved these changes Jan 22, 2020
View reviewed changes
Copy link
Member

@kszucs kszucs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wesm
Copy link
Member

wesm commented Jan 22, 2020

Are there still some problems?

@kou
Detect NEON availability
6395da2 
NEON isn't available on QEMU.
@kou
Copy link
Member Author

kou commented Jan 22, 2020

NEON is always enabled by #6042 but NEON isn't available on QEMU.

@cyb70289 Is 6395da2 acceptable?

We use QEMU to build .deb/.rpm packages for arm64 on x86_64 machine for now. We will use a real arm64 machine at Ursa Labs for .deb/.rpm packaging in the future but we don't set it up yet.

@kou
Copy link
Member Author

kou commented Jan 22, 2020

@ursabot crossbow submit -g linux-arm

@kszucs
Copy link
Member

kszucs commented Jan 22, 2020

AMD64 Conda Crossbow Submit (#87764) builder has been succeeded.

Revision: 6395da2

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-448

Task Status
centos-7-aarch64 Azure
centos-8-aarch64 Azure
debian-buster-arm64 Azure
debian-stretch-arm64 Azure
ubuntu-bionic-arm64 Azure
ubuntu-disco-arm64 Azure
ubuntu-eoan-arm64 Azure
ubuntu-xenial-arm64 Azure

@cyb70289
Copy link
Contributor

NEON is always enabled by #6042 but NEON isn't available on QEMU.

@cyb70289 Is 6395da2 acceptable?

We use QEMU to build .deb/.rpm packages for arm64 on x86_64 machine for now. We will use a real arm64 machine at Ursa Labs for .deb/.rpm packaging in the future but we don't set it up yet.

I'm okay with the changes.
In my understanding, armv8 64 bit compiler(aarch64) supports neon, no matter run by qemu or on real hardware. I see options like "-mfpu=neon-fp-armv8" which is aarch32 only options in this patch. Not sure if you're running armv8 32 bit compiler?

@kou
Remove needless NEON detection
16be900 
-mfpu=neon-fp-armv8 is for 32bit.
@kou
Use RAPIDJSON_SIMD instead of our ARROW_RAPIDJSON_SKIP_WHITESPACE_SIMD
705a53d 
RapidJSON 1.1.0 doesn't support NEON. Unreleased RapidJSON supports
NEON. We can detect whether using RapidJSON supports NEON or not by
RAPIDJSON_SIMD. RapidJSON 1.1.0 with RAPIDJSON_NEON doesn't define
RAPIDJSON_SIMD. Unreleased RapidJSON with RAPIDJSON_NEON defines
RAPIDJSON_SIMD.
@kou
Copy link
Member Author

kou commented Jan 23, 2020 •
edited

In my understanding, armv8 64 bit compiler(aarch64) supports neon, no matter run by qemu or on real hardware. I see options like "-mfpu=neon-fp-armv8" which is aarch32 only options in this patch. Not sure if you're running armv8 32 bit compiler?

Oh, I didn't know about it. Thanks.

Then we don't need NEON detection.

I found the cause of the build error. RapidJSON 1.1.0 doesn't have NEON support.

I changed to use RAPIDJSON_SIMD instead of our ARROW_RAPIDJSON_SKIP_WHITESPACE_SIMD. RAPIDJSON_SIMD isn't defined with RapidJSON 1.1.0 and RAPIDJSON_NEON. RAPIDJSON_SIMD is defined with unreleased RapidJSON and RAPIDJSON_NEON.

https://github.com/Tencent/rapidjson/blob/master/include/rapidjson/rapidjson.h#L351-L354

@kou
Copy link
Member Author

kou commented Jan 23, 2020

@ursabot crossbow submit -g linux-arm

@kszucs
Copy link
Member

kszucs commented Jan 23, 2020

AMD64 Conda Crossbow Submit (#87797) builder has been succeeded.

Revision: 705a53d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-449

Task Status
centos-7-aarch64 Azure
centos-8-aarch64 Azure
debian-buster-arm64 Azure
debian-stretch-arm64 Azure
ubuntu-bionic-arm64 Azure
ubuntu-disco-arm64 Azure
ubuntu-eoan-arm64 Azure
ubuntu-xenial-arm64 Azure

@kou
Copy link
Member Author

kou commented Jan 23, 2020

RapidJSON build error is fixed but 6h limit error of Azure Pipelines is caused.
The limit error is unrelated. So I'll merge this.

@kou kou closed this in d0e0e50 Jan 23, 2020
@kou kou deleted the packaging-linux-arm64 branch January 23, 2020 11:52
@ganeshrajuLinaro
Copy link

@kou Do you need an arm hardware (we could donate to setup CI) so you don't have to build under qemu?

@kou
Copy link
Member Author

kou commented Feb 10, 2020

It's helpful.
Let's discuss it on the dev@arrow.apache.org mailing list. Could you send an e-mail to dev@arrow.apache.org?

@wesm
Copy link
Member

wesm commented Feb 11, 2020

Yes, let's discuss on the mailing list. It looks like we be able to use Buildkite on ASF repos now (due to some permissions changes in GitHub's APIs) so having some dedicated ARM hardware available on BK (or GitHub Actions self-hosted, if that won't work) would work well. It is almost trivial for third parties to attach workers to a Buildkite instance

@ganeshrajuLinaro
Copy link

Sent email. We can surely provide dedicated ARM hardware.
@hrw can help with setting it up, if we get go-ahead.

@guyuqi guyuqi mentioned this pull request Feb 20, 2020
Closed
@asfimport asfimport mentioned this pull request Feb 19, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kszucs kszucs kszucs approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@kou @kszucs @wesm @cyb70289 @ganeshrajuLinaro