How can I get more information to help me deploy tapir?

[WeiMengXS]

When I try to upload the provider, it returns {"errorId":"41ede21f-72c9-4df3-a33b-e9f75eea3965","errors":[{"message":"An unexpected error has occurred. Please raise an issue if you think this is a bug."}]
But based on the existing documents, I haven't been able to find out where the problem lies?

Console output of tapir：

No further information...

[PacoVK]

Thanks for reporting! Could you please share further insights?
Ideally the request you made and the folder structure of your provider, if you tried to push a custom build one.

[WeiMengXS]

Hello paco!
Can you help me answer this question?

I got this error while uploading.
2023-12-25 10:10:16,416 SEVERE [api.map.exc.ThrowableMapper] (executor-thread-3) errorId e8cc5635-4546-4f64-8e88-b816564585a2: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /tmp/13909245068916160045/archive/terraform-provider-yunti_0.0.2_darwin_arm64.zip

I entered the container according to the path and found that the file looked like this:

Is there something wrong with my operation?

The directory structure I uploaded is like this:

Commands used:
curl -XPOST --fail-with-body -H 'x-api-key: xxxxxxxx' -F archive=@archive.zip "https://example.com/terraform/providers/v1/fc/yunti/v0.0.2"

[PacoVK]

Assuming that the archive size is 318804492 bytes (~319MB), did you already set API_MAX_BODY_SIZE to a custom value? Because default is 100MB.

[WeiMengXS]

Hello, I have set API_MAX_BODY_SIZE to 1GB, but I'm still encountering the same issue even though the file size being uploaded this time is around 50MB.
-e API_MAX_BODY_SIZE=1024M

Commands used:
curl -XPOST --fail-with-body -H 'x-api-key: aZAk6Y6Tu8PhLmcYZnaUN7M7' -F archive=@archive.zip "https://example.com/terraform/providers/v1/fc/yunti/v0.0.2"
Run:

docker run -d --name tapir   -p 8088:8080  
-e API_MAX_BODY_SIZE=1024M 
-e BACKEND_CONFIG=elasticsearch   
-e BACKEND_ELASTICSEARCH_HOST=xxxx:9200  
 -e STORAGE_CONFIG=host   
-e REGISTRY_HOSTNAME=localhost  
 -e REGISTRY_PORT=443 
-e REGISTRY_GPG_KEYS_0__ID=xxxx 
-e REGISTRY_GPG_KEYS_0__ASCII_ARMOR=xxxxx 
-e AUTH_ENDPOINT=http://xxxxx:8080/auth/realms/terraform 
-e AUTH_CLIENT_ID=registry 
pacovk/tapir

[PacoVK]

Ah I think the docs could be more expressive, for the STORAGE_CONFIG.
You set STORAGE_CONFIG=host which is not valid, to use local volume please set to STORAGE_CONFIG=local

Let me know if that fixes the issue, I will extend the startup script to also check for the storage config and print a better Error message in case it is invalid.

[WeiMengXS]

Thank you for your reply.
Oh, sorry for the mistake. However, I have already set STORAGE_CONFIG to local, but the issue still persists. Are there any other directions for troubleshooting?

 docker run -d --name tapir   -p 8088:8080  
-e API_MAX_BODY_SIZE=400M 
-e BACKEND_CONFIG=elasticsearch   
-e BACKEND_ELASTICSEARCH_HOST=xxx:9200   
-e STORAGE_CONFIG=local   
-e REGISTRY_HOSTNAME=localhost   
-e REGISTRY_PORT=443 
-e REGISTRY_GPG_KEYS_0__ID=xxx 
-e REGISTRY_GPG_KEYS_0__ASCII_ARMOR=xxx 
-e AUTH_ENDPOINT=http://xxxx:8080/auth/realms/terraform 
-e AUTH_CLIENT_ID=registry 
pacovk/tapir

Commands used:

$ curl -XPOST  -H 'x-api-key:aZAk6Y6Tu8PhLmcYZnaUN7M7' --fail-with-body  -F archive=@archive.zip https://examlpe.cn/terraform/providers/v1/fc/yunti/v0.0.2
curl: (22) The requested URL returned error: 500
{"errorId":"53b38893-a350-4027-b1ba-81a3a9894435","errors":[{"message":"An unexpected error has occurred. Please raise an issue if you think this is a bug."}]

My file:

The backend files are:

[PacoVK]

I will try to reproduce and investigate a bit further. Thanks for the comprehensive report👌

[PacoVK]

@WeiMengXS i tried to write a reproducer with the following:
ℹ️ For Keycloak i used my dev setup, but that should not have any influence, since you are already able to login and create the deploykey.
ℹ️ I also did no setup any GPG_ things because you wrote the error occures during upload of the provider. I omitted any setting that is relevant for downloading providers to keep it simple

docker-compose.yml

version: '3'
services:

  tapir:
    image: pacovk/tapir
    ports:
      - "8088:8080"
    environment:
      API_MAX_BODY_SIZE: 400M
      BACKEND_CONFIG: elasticsearch
      BACKEND_ELASTICSEARCH_HOST: opensearch-node1:9200
      STORAGE_CONFIG: local
      AUTH_ENDPOINT: http://keycloak:8080/realms/tapir

  keycloak:
    image: quay.io/keycloak/keycloak:22.0
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./dev/scripts/keycloak:/opt/keycloak/data/import
    ports:
      - "8080:8080"
    command:
      - start-dev --import-realm
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin

  opensearch-node1:
    image: opensearchproject/opensearch
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - plugins.security.disabled=true
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.type=single-node
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    ports:
      - "9200:9200" # REST API
      - "9600:9600" # Performance Analyzer

I used a fake provider as demo with similar folder structure as you described
I did then the following:

1. create a deploykey via Tapir for the provider, where namespace=fc and type=yunti
2. issue a upload command to Tapir via curl curl -XPOST -H 'x-api-key:tM6pX3IlTBROmLAJk8Nm4Fhy' --fail-with-body -F archive=@Archiv.zip http://localhost:8088/terraform/providers/v1/fc/yunti/v0.0.2
3. No error...

I used the following fake provider source:
Archiv.zip

How to proceed?

• verify you used the latest Tapir version (to be sure, run docker pull pacovk/tapir)
• Increase the log level by setting envinroment variable QUARKUS_LOG_LEVEL=debug
• Please check if you can upload the "fake provider" i used here.
• If not, please also check if you have any extra setting for Tapir set in your container, apart from those you already posted.
• For further investigation, please try to setup a local Tapir instance from the docker compose reference aforementioned and try to upload your provider there (mind without SSL and GPG you wont be able to download but it should at least work with the upload
• Keep me posted here, in case i can further assist i am happy to do so

[WeiMengXS]

Thank you very much for this detailed guidance. I have been able to successfully deploy and publish my provider!

[WeiMengXS]

Initializing the backend...

Initializing provider plugins...

• Finding latest version of xxx.com.ngrok-free.app/fc/yunti...
• Installing xxx.com/fc/yunti v0.0.2...
  ╷
  │ Error: Failed to install provider
  │
  │ Error while installing xxx.com/fc/yunti v0.0.2: checksum list has no SHA-256 hash for "terraform-provider-yunti_0.0.2_darwin_arm64.zip"
  ╵
  But I encountered this problem. I debugged and found that it seemed to be an auth problem with keylocak.

[PacoVK]

I think that is more about the provider artefact. You strictly have to fulfill the requirements described in the official Hashicorp docs

In your case the SHASUM file might not be correct. Read here if you try to manually prepare the release. The GPG key used for signing must be the same passed to Tapir!

[WeiMengXS]

Hashicorp 官方文档I am quite familiar with it now, it seems that when TF is accessing the backend, it is redirected to the Keycloak's verification URI. Do you have any reference for setting up Keycloak? Because I am not very familiar with Keycloak itself, This is a bit confusing for me.

"_NOTE: To use Tapir UI you need to be authenticated. However, you can read the registry without authentication. In particular the Terraform CLI will work without authentication"
In other words, how to implement this.

[PacoVK]

You found a bug with the local storage backend, I will try to fix it soon. The storage api needs to be accessible without authentication to allow read from the registry! No issue with Keycloak. Thanks for sharing this!

[WeiMengXS]

When is it expected to be restored? I'm very interested in this project!

[PacoVK]

I already fixed it and will create a release probably today, latest tomorow.

[PacoVK]

Here we go 0.6.2 contains the fix https://github.com/PacoVK/tapir/releases/tag/0.6.2

[WeiMengXS]

I have verified it and there seems to be no issue. Thank you very much.
However, it seems that there could be room for improvement in displaying it on the frontend,I'm not sure if there is a misconfiguration somewhere.

If this is a domain name, should the 'https://' prefix actually be removed? In Terraform, it is expected to follow the format [hostname/][namespace/]name. The "source" attribute must be in the format "[hostname/][namespace/]name"

[PacoVK]

Good point, I'll add that on the list. Actually it makes no difference but without the protocol it follows the naming of hashicorp in a consistent way 👌
Thanks again for the valuable feedback!

[PacoVK]

@all-contributors please add @WeiMengXS for bug, ideas

[allcontributors]

@PacoVK

I've put up a pull request to add @WeiMengXS! 🎉