Agent wont start after update "Unable to retrieve local agent IP address"

[Trufax]

Hello,

i use watchtower to update my containers but the Portainer agent keeps to have issues when updated.
Whenever watchtower updates the container, it keeps restarting with the following error message:

agent_1 | 2021/08/05 12:22:25 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: Error: No such container: 6c9fd4504fb5]

And i use the following compose file:

portainer:
    image: portainer/portainer-ce
    environment:
      # REQUIRED: Should be equal to the service name prefixed by "tasks." when
      # deployed inside an overlay network
      AGENT_SECRET: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    ports:
      - "9000:9000"
      - "8000:8000"
    volumes:
      - portainer_data:/data
    restart: unless-stopped
      
  watchtower:
    image: v2tec/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: portainer_agent_1 portainer_watchtower_1
    restart: unless-stopped

I can fix it by manually running docker-compose down and than up again but this defeats the whole purpose of the automatic updates.

Regards

[jackman815]

I had the same problem around 3 months.

[Trufax]

I guess you still have the issue now or did you solve it somehow ?

[jackman815]

I still have this issue, I just recreate it every time

[jackman815]

Maybe set the watchtower to monitor only mode for the portainer-agent container.

Add the label com.centurylinklabs.watchtower.monitor-only to portainer-agent

[jackman815]

# docker inspect fc8934356e2d | grep Hostname
            "Hostname": "c77766f05c0e",

I found the problem is about the watchtower, it clones the container configs included hostname when upgrading the container.

When upgrade/re-create the container, by default the docker daemon will assign a new hostname to the container and update internal dns, but the watchtower won't update the container hostname after the container upgrade so the agent will try to look up the old hostname and it would be getting nxdomain result by docker internal dns.

agent/os/hostname.go

Line 6 in b39a072

return os.Hostname()

https://github.com/containrrr/watchtower/blob/9bb8991a768b5d0071bd69f878b47ef3be513311/internal/actions/mocks/container.go#L77

// CreateMockContainerWithConfig creates a container substitute valid for testing
func CreateMockContainerWithConfig(id string, name string, image string, running bool, restarting bool, created time.Time, config *dockerContainer.Config) container.Container {
	content := types.ContainerJSON{
		ContainerJSONBase: &types.ContainerJSONBase{
			ID:    id,
			Image: image,
			Name:  name,
			State: &types.ContainerState{
				Running:    running,
				Restarting: restarting,
			},
			Created: created.String(),
			HostConfig: &dockerContainer.HostConfig{
				PortBindings: map[nat.Port][]nat.PortBinding{},
			},
		},
		Config: config,
	}
	return *container.NewContainer(
		&content,
		&types.ImageInspect{
			ID: image,
		},
	)
}

[jackman815]

I try to add the --hostname arg to start the container, waiting for the next agent's update and check the working status.

My config:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v portainer_agent_data:/data \
  --network="bridge" \
  --hostname portainer_edge_agent \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID= \
  -e EDGE_KEY= \
  -e CAP_HOST_MANAGEMENT=1 \
  --name portainer_edge_agent \
  portainer/agent

[Trufax]

Thanks for sharing. I added

hostname: portaineragent
container_name: portaineragent

to my compose file, lets see if this works with the next update.

[Trufax]

i can confirm that setting the hostname and the container_name fixed the issue for me. Portainer agent is now updating fine with watchtower.

[GiuffreLab]

Was able to fix this from Portainer by just stopping continer > Duplicate / Edit > scroll down to advanced container settings > Network > make hostname field match container name > Deploy Container