Annoy build index failing in docker image built from Github Actions

[williambrach]

Hi!

We tried to implement Annoy angular index into our FastAPI python service which runs on Azure k8s service.

An index is being built every time from the pandas dataframe when the service starts. The index build code is snipped below.

Data from which we build the index has 10k rows and a float vector of size 9.

n_trees = 10
f = 9
ANNOY_INDEX = AnnoyIndex(9, "angular")
for i, row in dfMl.iterrows():
  wineId = int(row[:1].values[0])
  wineValues = row[1:].values
  ANNOY_INDEX.add_item(wineId, wineValues)
  
ANNOY_INDEX.build(n_trees, n_jobs=1)

Locally on Apple M1 machine, the service works fine, and when we build a docker image locally and push it to ACR from which we deploy the image to k8s, everything works too.

docker buildx build --platform linux/amd64 -t apiservice:1.0.0 .
docker push apiservice:1.0.0

The issue appears when we try to build the image with the GitHub Actions pipeline (docker push and build). The build will not fail and we are able to deploy image built in GitHub actions to our container registry. But when we try to deploy this image into k8s, the pod fails with the status CrashLoopBackOff.

In the GitHub actions pipeline, we specify the platform as linux/amd64 same as when we are building the image locally.

From the pod description, we can get this exit code - 132 which refers to an unsupported processor instruction.

Last State:     Terminated                                                                                                                                                                                                                     
   Reason:       Error                                                                                                                                                                                                                         
   Exit Code:    132

We tried to log this issue and found out the service will fail at index.build().

Did someone encounter a similar issue or could help us figure this one out?

Thanks!

[erikbern]

You might have to build it with AVX instructions disabled – I forget how, but maybe try to enable the NO_MANUAL_VECTORIZATION definition

[williambrach]

How should I pass NO_MANUAL_VECTORIZATION parameter to annoy? Should I export it as an env variable?

When we want to build annoy with AVX instruction disabled should we build it from the source or there is a release version without these instructions?

[erikbern]

I think you might be able to do something like this (I'm just looking at setup.py right now)
ANNOY_COMPILER_ARGS=-DNO_MANUAL_VECTORIZATION when annoy is installed. But I'm not sure!

[williambrach]

Thanks, I will try to pass these flags as environment variables before the pip install.

[SteliosGian]

We're experiencing the same issue when we install annoy via pip in docker image using python 3.8.15. Using the option ANNOY_COMPILER_ARGS=-DNO_MANUAL_VECTORIZATION looks to solve the issue.
Is there any impact in functionality? I'm asking because I see this line in setup.py file.
https://github.com/spotify/annoy/blob/master/setup.py#L67

[SteliosGian]

Some more info on that, it looks like the issue comes from the wheel. When we build the wheel locally and just upload it to our repo it works fine.

[williambrach]

@SteliosGian you suggest a workaround for this issue, to build Annoy locally and upload the built package to your GitHub repository. Then, you can run GitHub's CI/CD to deploy?

[SteliosGian]

Yes, and normally install the wheel with pip (e.g., pip install annoy-1.17.1-cp38-cp38-linux_x86_64.whl).

The second workaround is to set the env variable in docker mentioned above just before the annoy installation.

e.g., ENV ANNOY_COMPILER_ARGS=-DNO_MANUAL_VECTORIZATION