Support ppc64le OpenSearch distribution for POWER architecture.

[SonalMahambrey1]

Opensearch snapshot for X64 is available at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
As power architecture is supported now, any update when the similar artifacts will be published for ppc64le as well ?

[dblock]

I am not sure whether the build team intends to publish these, or whether the team at IBM (cc: @seth-priya) wants to maintain this port, but let's move this into opensearch-build.

[seth-priya]

@dblock it would be great if the build team can publish these and we will continue to watch for Power specific issues and fix them, on our side.

[dblock]

@bbarani @peterzhuamazon what would it take to add those to automation? Maybe @seth-priya can contribute?

[zelinh]

[Triages] First step is to have the IBM server in order to move forward. Adding @elfisher and @setiah for discussion.

[seth-priya]

There are a few places to get access to ppc64le VMs, one of them is https://osuosl.org/services/powerdev/request_hosting/

[Sunidhi-Gaonkar1]

Hi Team, any thoughts on above?

[jordarlu]

Adding @bbarani to comment on future supporting PowerPC 64 Little Endian ..thanks!

[seth-priya]

thanks @jordarlu , we can definitely contribute the changes that are needed for building the artifacts via automation and we can also discuss about the infrastructure that's needed for the same (if needed on a call?). As of now - we already have the Power changes up-streamed for the main opensearch repository and have been able to locally build the images and test them and also have a working build and image for opensearch-dashboards. Currently, we are building/validating all of the opensearch plugins separately on Power to make sure all of them work smoothly (though from what we have seen so far, they should). I am also adding our overall technical lead/architect @gerrith3 here.

[gerrith3]

Yeah - we would love to help support getting a CI environment set up for ppc64le. I'd love to know what the options are from your side, what we would need to integrate with, etc. We have some challenges with broadly available CI platforms but we do have several solutions, alternatives, and work arounds to integrate with most/many environments.

[Sapana-Khemkar]

@dblock, @bbarani, @jordarlu and others,
I am from @seth-priya's team. We can help adding/maintaining ppc64le support to CI environment. Let us know how can we
move further on this.

[bbarani]

@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.

[Sapana-Khemkar]

@Sapana-Khemkar Thanks for your support. How do you want to proceed with next steps? Would a kick-off call help to get things started soon? It would be really easy to collaborate via OpenSearch Slack workspace.

@bbarani joined slack and sent you message on devops channel.

[prudhvigodithi]

Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports ppc64le, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside the opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 docker) we discussed over the call?
Thank you

[gerrith3]

Hey for the OpenSearch folks, if you do request a free VM at OSL per the link earlier in this thread, reference my name as your IBM support contact and we will make sure it gets expedited for you. We would love to see this integrated and we encounter more users of OpenSearch on Power all the time, so this would be a great step for all of us!!

[gerrith3]

And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)

[Sunidhi-Gaonkar1]

Hi @prudhvigodithi , I am from @seth-priya and @Sapana-Khemkar's team. I was able to manually execute the build and assemble workflows on Power server and create an artifact for ppc64le.
The docker image opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 is not available for power, hence was not able to execute workflows inside the container.
There were a few changes required to be made. I have attached logs of output for execution of build and assemble workflows, also attached the changes that were made to build the artifact.

OpenSearch_assemble.txt
OpenSearch_build.txt
OpenSearch_Patch.txt

[bbarani]

And, apologies if this should be in a thread of its own, but does anyone know if OpenSearch is keeping up with Elasticsearch's ability to be used as a vector database? I've been seeing that question come up several times and at the moment, I have no clue. ;) thanks, and apologies for hijacking another thread, although this is really yet another Power use case for us. ;)

OpenSearch indeed has made great strides in Vector DB space. It uses K-NN functionality as documented in this blog to implement vector db functionality. The issues related to Vector DB functionalities are tracked here.

[Sapana-Khemkar]

Had an offline sync up with @Sapana-Khemkar, since OpenSearch supports ppc64le, having a Power9 server and running the build and assemble workflow should give us an artifact for ppc64le. While we request for the server and integrate with the CI/CD @Sapana-Khemkar can you please post the results of the steps (manually executing the workflows on your server inside the opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 docker) we discussed over the call? Thank you

@prudhvigodithi any update from your side? Have you raised request for ppc64le vm?

[prudhvigodithi]

Hey @Sapana-Khemkar I have the form submitted NAME: OpenSearch Project, support.osuosl.org #33270.
Thank you
@bbarani @peterzhuamazon

[Sapana-Khemkar]

Hey @Sapana-Khemkar I have the form submitted NAME: OpenSearch Project, support.osuosl.org #33270. Thank you @bbarani @peterzhuamazon

@prudhvigodithi your request 33270 is approved. Please let me know if you face any issue.

[prudhvigodithi]

Thanks, @Sapana-Khemkar what is the next step here? I just got an email as Approved for the request I created. Let me know we can discuss the next steps in public slack on how to securely connect the ppc64le VM as jenkins agent.

[peterzhuamazon]

Have a discussion with @prudhvigodithi and we need to add a test ci-runner image layer of linux/ppc64le, then try to start it up on the requested server before trying out the built.

If successful, then we will proceed with a manual build, before adding the changes to all the ci-runner images.

I will start on creating the new ci-runner image soon.

[Sapana-Khemkar]

1. ml-commons - we are working on this.
2. Official support from gh-cli

We have raised PR for ppc64le support. But it may take a little time. Meanwhile can you install go v1.21.0 using below steps

wget https://go.dev/dl/go1.21.0.linux-ppc64le.tar.gz
tar -C /usr/local -xzf go1.21.0.linux-ppc64le.tar.gz
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin:/usr/local/bin
go version

4. @Sunidhi-Gaonkar1 will raise PR for build.sh and update you. For changes in Jenkins file we will wait for your PR to get merged.
5. @Sunidhi-Gaonkar1 will be looking into pytest errors due to PyYAML 6.0.1.
6. For dynamic provision of ppc64le servers, you can raise request with OpenStack Access. While raising vm request select "I'do like to have access to the Openstack GUI/API" option (as highlighted in below image) and use Openstack APIs for dynamically provision ppc64le servers.

[peterzhuamazon]

Thanks @Sapana-Khemkar ,

On 7, @prudhvigodithi has completed the review of my ci-runners changes.

Since building these images takes a lot of time, so I plan to reduce the load if possible.

This PR is in review to add gh cli ppc64le support, and I will give it a few days to see gh team reply.

• Add ppc64le support. cli/cli#8612

If their review and support takes a lot of time, I will merge #4336 and start building the images next week.

Thanks.

[peterzhuamazon]

On 6, I am still a bit confused as that seems like still manual approaches to request servers, without internally connect through EC2 APIs.

Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.

[seth-priya]

Or is there is any plugin that can directly connect Jenkins provision agents with openstack, we still need to look into that in details.

@peterzhuamazon yes, there is an openstack plugin for jenkins that can help with dynamic provisioning of VMs once you have openstack access - https://plugins.jenkins.io/openstack-cloud/ , https://plugins.jenkins.io/openstack-cloud/releases/

[peterzhuamazon]

Sent OpenStack Request.

[peterzhuamazon]

Due to #4241 #4370, we will delay the image generation for ppc64le, to unblock 2.12 release issues with IntegTests.

[Sunidhi-Gaonkar1]

@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command pipenv run isort . as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.

[peterzhuamazon]

@peterzhuamazon, I was able to resolve the pytest errors faced for #4348. The errors were thrown due to code linting issue. The sorting order was fixed after running command pipenv run isort . as mentioned in https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#code-linting.

Hi @Sunidhi-Gaonkar1 if you resolve the sort issue and actually run pipenv run pytest you will see new issues. The isort issues is expected and not related to the pytest issues.

Thanks.

[peterzhuamazon]

Receive credentials on accessing to OpenStack and its endpoints. Will work on connecting to Jenkins.

[Sunidhi-Gaonkar1]

Raised PR #4413 to fix pytests and update build.sh to add ppc64le build target.
Please review, Thank you.

[peterzhuamazon]

Use package manager for go installation for easier setups.
Move gh to https://github.com/cli/cli/releases/tag/v2.32.1 since that is the last version using go < 1.21.

Update: Not possible:

go install github.com/cli/cli/v2/cmd/gh@v2.32.1
go: downloading github.com/cli/cli/v2 v2.32.1
go: github.com/cli/cli/v2/cmd/gh@v2.32.1 (in github.com/cli/cli/v2@v2.32.1):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

[peterzhuamazon]

Only possible when manually checkout the repo then make:

make install prefix=/path/to/gh

[peterzhuamazon]

The 1st docker ci images with ppc64le layer is up online: https://hub.docker.com/r/opensearchstaging/ci-runner/tags?page=1&ordering=last_updated&name=ci-runner-centos7-opensearch-build-v3.8

More to come.

[peterzhuamazon]

More issues:

Rockylinux8:

ERROR: failed to solve: rockylinux:8: no match for platform in manifest: not found

[peterzhuamazon]

We might need to change to almalinux from rockylinux due to version 8 is not having ppc64le layer for rockylinux.

[peterzhuamazon]

More to read:
https://rockylinux.org/news/rocky-linux-9-2-power-le-update/

[peterzhuamazon]

All build and test ci images now support ppc64le and have gh installed with go modules.
However, docker release images will not support ppc64le unless AmazonLinux 2023 image start supporting it.

[Sapana-Khemkar]

All build and test ci images now support ppc64le and have gh installed with go modules. However, docker release images will not support ppc64le unless AmazonLinux 2023 image start supporting it.

Thanks @peterzhuamazon please go ahead with this part and release tar for power. We will work with you on docker images separately.

[peterzhuamazon]

We will continue with the Almalinux switch after 2.13.0 release.

[Sunidhi-Gaonkar1]

We will continue with the Almalinux switch after 2.13.0 release.

Hi @peterzhuamazon , Any updates on this?