Redeployment with changes in service definition alters allocation IP address (multiple interfaces)

[ahaczewski]

Nomad version

$ nomad version
Nomad v1.0.3 (08741d9f2003ec26e44c72a2c0e27cdf0eadb6ee)

Operating system and Environment details

$ uname -a
Linux node2 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

System: Ubuntu 18.04.05 LTS

Driver: raw_exec

Rootless Nomad deployment, i.e., Nomad Client runs as a non-root user.

Issue

Redeployment of a job with changes to service causes allocation to acquire IP address of a different, public network interface, instead of the previously used private interface.

Strangely, when I redeploy the job with changes to the driver command's arguments, the allocation gets assigned the correct IP address from the private interface.

The problem is manifested with service scheduler, system is not affected.

NOTE

I looked at the other issue #10010 and although it looks similar, I think the problem is different.

Reproduction steps

1. Deploy the job and wait until it's healthy.
2. Confirm the allocation has a private address.
3. Add a tag to the service definition in the .nomad file.
4. Redeploy the job.
5. Note that the new allocation now has a public address.

Job file (if appropriate)

Please note that the job below is not the original one, but we're able to replicate the problem in Vagrant VM with this simple job as well.

job "test-network-service" {
  type = "service"
  datacenters = ["dc1"]
  group "test-network-service" {
    count = 4
    network {
      port "http" {
        static = 9020
      }
      port "http2" {
        static = 9022
      }
    }
    service {
      name = "test-network-service"
      port = "http"
      tags = ["tag3"]
    }
    service {
      name = "test-network-service2"
      port = "http2"
      tags = ["tag3"]
    }
    task "test-network-service" {
      driver = "raw_exec"
      config {
        command = "bash"
        args = ["-c", "echo ${NOMAD_IP_http} ; tail -f /dev/null"]
      }
      resources {
        cpu    = 10
        memory = 10
      }
    }
  }
}

Job status prior to deployment

$ nomad job status webapp
ID            = webapp
Name          = webapp
Submit Date   = 2021-02-11T13:55:53Z
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
webapp      0       0         1        23      17        0

Latest Deployment
ID          = 59e007f1
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
webapp      1        1       1        0          2021-02-11T14:06:35Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created     Modified
0d2bbaec  e3ee2eac  webapp      32       run      running   6m20s ago   5m37s ago

Alloc status prior to deployment

$ nomad alloc status 0d2bbaec
ID                  = 0d2bbaec-31fe-9095-b2a3-3906766d41f3
Eval ID             = 9b41f656
Name                = webapp.webapp[0]
Node ID             = e3ee2eac
Node Name           = node2
Job ID              = webapp
Job Version         = 32
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 19m52s ago
Modified            = 19m9s ago

Allocation Addresses
Label  Dynamic  Address
*http  yes      10.8.0.2:24410

Task "webapp" is "running"
Task Resources
CPU         Memory           Disk     Addresses
67/400 MHz  1.5 GiB/1.0 GiB  300 MiB

Task Events:
Started At     = 2021-02-11T14:19:46Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2021-02-11T14:19:46Z  Started                Task started by client
2021-02-11T14:19:45Z  Downloading Artifacts  Client is downloading artifacts
2021-02-11T14:19:45Z  Task Setup             Building Task Directory
2021-02-11T14:19:45Z  Received               Task received by client

Nomad Client configuration

[,..]
client {
    network_interface = "10.8.0.2"

    host_network "public" {
      interface = "<public_ip>"
    }
}
[,..]

Plan

+/- Job: "webapp"
+/- Task Group: "webapp" ( 1 in-place update )
  +/- Task: "webapp" ( forces in-place update )
    +/- Service {
      AddressMode: "auto"
      EnableTagOverride: "false"
      Name: "webapp"
      PortLabel: "http"
      TaskName: ""
      + Tags {
        + Tags: "test"
          Tags: "http"
          Tags: "traefik.enable=true"
          Tags: "traefik.http.routers.webapp.rule=Host(`public.domain.com`) && PathPrefix(`/rest/webapp`)"
          Tags: "traefik.http.routers.webapp.service=webapp"
        }
      }

Job status after deployment

$ nomad job status webapp
ID            = webapp
Name          = webapp
Submit Date   = 2021-02-11T15:09:01Z
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
webapp      0       0         1        23      18        0

Latest Deployment
ID          = 5c526ec9
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
webapp      1        1       1        0          2021-02-11T15:19:12Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created     Modified
83cff3b7  97912218  webapp      34       run      running   3m11s ago   19s ago
0d2bbaec  e3ee2eac  webapp      32       stop     complete  49m46s ago  3m7s ago

Alloc status after deployment

$ nomad alloc status 83cff3b7
ID                  = 83cff3b7-000b-edd3-0f8c-9e3996d7e598
Eval ID             = 9c7ecaa1
Name                = webapp.webapp[0]
Node ID             = 97912218
Node Name           = node3
Job ID              = webapp
Job Version         = 34
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 3m21s ago
Modified            = 29s ago
Deployment ID       = 5c526ec9
Deployment Health   = healthy

Allocation Addresses
Label  Dynamic  Address
*http  yes      <public_ip>:24410

Task "webapp" is "running"
Task Resources
CPU        Memory           Disk     Addresses
0/400 MHz  1.4 GiB/1.0 GiB  300 MiB

Task Events:
Started At     = 2021-02-11T15:06:26Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type                   Description
2021-02-11T15:09:01Z  Task hook failed       consul_services: error getting address for check "http": invalid port "http": port label not found
2021-02-11T15:06:26Z  Started                Task started by client
2021-02-11T15:06:24Z  Downloading Artifacts  Client is downloading artifacts
2021-02-11T15:06:24Z  Task Setup             Building Task Directory
2021-02-11T15:06:21Z  Received               Task received by clien

NOTE

The allocation was done on another server (node3) instead of the original (node2).

[mr-karan]

@ahaczewski Looks similar to what I was facing in #10001

[ahaczewski]

@mr-karan Yes, looks VERY similar. Due to how the redeployment behaves where there is a change just to service stanza, and that system scheduler works correctly, the suspect is the problem with service scheduler or somewhere there.

I'll try to have a look at the code myself, maybe I'll be able to spot something.

[AndrewChubatiuk]

@ahaczewski
You can try a latest build from master, looks like this issue was fixes already since 1.0.3 in this commit

[ahaczewski]

@AndrewChubatiuk Thanks! I will try tomorrow and will get back to you with a feedback.

[AndrewChubatiuk]

@tgross
Is it worth having a separate code for SystemScheduler instead of merging it with GenericScheduler? There are a lot of code duplications and looks like it causes problems like this from time to time

[AndrewChubatiuk]

@ahaczewski
Actually I'm not a maintainer, just curious, cause using this functionality as well and looking forward for a new release to fix issues that I'm facing in my project:)

[ahaczewski]

The bug is fixed in master. When is the release planned? 😅

[tgross]

Hi @ahaczewski I can't promise anything or give an exact day but Nomad 1.0.4 will likely ship this week.

[ahaczewski]

@tgross Awesome, thank you very much for the heads up.