Using Podman as docker substitute with terraform does not work as expected

[2fxprogeeme]

Issue Description

I use the following Podman version on Linux Mint /Ubuntu based)

Client: Podman Engine
Version: 4.6.2
API Version: 4.6.2
Go Version: go1.18.1
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64

Podman can be activated via systemctl socket (unix:///run/user/1000/podman/podman.sock). This was tested with docker-compose.

Question: Is it a valid scenario to substitute docker with podman when using terrafom? If so, is the behavior I described an error or do I have to change any configuration settings in my environment?

Regards
Ulrich

Steps to reproduce the issue

See ecplanation

Describe the results you received

Now I want to use terraform with docker provider to handle some containers. For a 1st test I created a small terraform file to handle a NGINX Container

terraform {
    required_providers {
        docker = {
            source  = "kreuzwerker/docker"
            version = "3.0.2"
        }
    }
}
provider "docker" {
      host = "unix:///run/user/1000/podman/podman.sock"
}
resource "docker_image" "nginx" {
      name         = "nginx:latest"
      keep_locally = false
}
resource "docker_container" "nginx" {
    image = docker_image.nginx.image_id
    name  = "tutorial"
    stop_timeout = 30
    ports {
        internal = 80
        external = 8080
    }
}

After putting this terraform file in an empty directory I execute the following terraform commands:

terraform init // Executed without error
terraform validate // No validation errors
terraform apply

I apply my changes and the NGINX container is executed. So far, so good. Executing "podman ps -a" shows:

ONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS        PORTS                 NAMES
f35217d2fedc  docker.io/library/nginx:latest  nginx -g daemon o...  4 seconds ago  Up 4 seconds  0.0.0.0:8080->80/tcp  tutorial

If I now execute "terraform destroy" and confirm with "yes" I get the following error message:

ocker_container.nginx: Destroying... [id=f35217d2fedcaef89592ceb16b3ce5ec139002577be43b2ea78c140275f79714]
╷
│ Error: Error waiting for container removal 'f35217d2fedcaef89592ceb16b3ce5ec139002577be43b2ea78c140275f79714': Error response from daemon: no such container
│ 
│ 
╵

The NGINX container is not longer running nor is it existing any more. "podman ps -a" shows an empty container list. But the NGINX container image is still there. "podman images shows"

REPOSITORY               TAG         IMAGE ID      CREATED      SIZE
docker.io/library/nginx  latest      bc649bab30d1  2 weeks ago  191 MB

If I now execute "terraform destroy" again, the image is also deleted.

Describe the results you expected

Container Image and Container itself could be managed using terraform and terraform docker provider.

podman info output

See explanation

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[flouthoc]

I suspect terraform is using API endpoint which is not correctly implemented in podman ( or shows inconsistent output as compared to docker ), running podman-server on debug mode can easily show which API is terraform trying to use for destroying container and we can match its consistancy with docker. Can you please show debug output of podman-server (service) ?

OTOH off-topic there are also third-party dedicated podman providers for terraform.

[2fxprogeeme]

I attach log output of podman.socket service. To produce the log, I executed the following terraform commands:

terraform init (no communication with podman I assume)
terraform validate (no communication with podman I assume)
terraform apply
terraform destroy (produces error message and keeps container image but removes container)
terraform destroy (destroys container image, no more error message)

podman.socket.log.txt

[flouthoc]

Okay something is going off when terraform makes a DELETE request just after STOPPING a container ( i.e container is already removed by the STOP operation so DELETE fails) . I'll try to install terraform and reproduce this on my local and report back.

[flouthoc]

I am able to reproduce but afaics the terraform provider attempting to perform wait request on container after performing delete. I'd suggest creating an issue with terraform provider itself.

[2fxprogeeme]

Thanks for investigation. I will continue with this issue on the terraform docker provider as you suggested.

[2fxprogeeme]

One last comment. I found this issue on terraform docker provider: kreuzwerker/terraform-provider-docker#584

Could this be the root cause for this issue here?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.