Test job specs with replicated jiva container(s) w.r.t their placements

[AmitKumarDas]

Assumptions:

• Nomad setup in a datacenter with 1 Server & 2 Clients

Steps:

• Create a job specs with 2 tasks
• 1 task will be for jiva
• 2nd task will be for jiva replica
• Each task will reside in different task groups
• Place appropriate constraints that enable placement of these tasks on different nodes

[AmitKumarDas]

Some more test scenarios w.r.t a predefined Nomad cluster setup:

Scenario 1:

• 2 client nodes
• One of the client node (out of 2 client nodes) will exhaust its resources
• A spec that defines 1 jiva main & 1 jiva replica
• The spec mandates jiva containers (main & replica) to run on separate nodes
• RESULT - Failed
• STATUS - Success

Scenario 2:

• 3 client nodes with unlimited resources to be used
• The spec should have 1 main jiva & 2 jiva-replicas
• The 2 jiva-replicas should not run on same node
• RESULT - Nomad will succeed in placement
• STATUS - Did not try out, but should work, considering success of other scenarios !!!

Scenario 3:

• 1 client node
• The spec should have 1 main jiva & 2 jiva-replicas
• The 2 jiva-replicas should not run on same node
• RESULT - Nomad will fail to place
• STATUS - Success

Scenario 4:

• 3 client nodes with unlimited resources to be used
• The spec should have 1 main jiva & 2 jiva-replicas
• The 2 jiva-replicas should not run on same node
• The main jiva's node will be shutdown
• The main jiva container should move to a running node
• RESULT - Nomad will succeed in placement
• STATUS - Success

Scenario 5:

• 3 client nodes with unlimited resources to be used
• The spec should have 1 main jiva & 2 jiva-replicas
• The 2 jiva-replicas should not run on same node
• The jiva replica 1's node will be shutdown
• The jiva replica 1 should not move
• RESULT - Passed
• STATUS - Success

[AmitKumarDas]

These are some observations or results. These may or may not be mentioned as scenarios in previous comment !!!

Observation 1

• Assumption: 1 Nomad server & 2 Nomad clients.
• Mark - the count property & the constraint distinct_hosts

# Note - this is just a snippet of the specs 
group "openebs" {	
   count = 5	  
   constraint {
      operator  = "distinct_hosts"
      value     = "true"
    }

• Result:

Task Group "openebs" (failed to place 3 allocations):
      * Constraint "distinct_hosts" filtered 2 nodes
    Evaluation "dbe69b48" waiting for additional capacity to place remainder

Observation 2

• The constraint "distinct_hosts" can be written following way:

constraint {
      operator  = "distinct_hosts"
}

[AmitKumarDas]

This is the observation for Scenarios 4 & 5 (mentioned at #2 (comment))

NOTE: Notice the movement of jiva controller container when its host goes down.
NOTE: Notice that jiva replica container does not move when its host goes down.
NOTE: Notice that jiva replica container comes up when its host comes up.

vagrant@client-01:~$ nomad status ctl
ID          = ctl-reps-jiva-vol
Name        = ctl-reps-jiva-vol
Type        = service
Priority    = 50
Datacenters = dc1
Status      = running
Periodic    = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
jiva-ctlg   0       0         1        0       0         0
jiva-repg   0       0         2        0       0         0

Allocations
ID        Eval ID   Node ID   Task Group  Desired  Status   Created At
21b74737  be3c7310  dbfbd795  jiva-repg   run      running  11/25/16 09:20:04 UTC
803073c1  be3c7310  dc3199f3  jiva-repg   run      running  11/25/16 09:20:04 UTC
b109dd93  be3c7310  dbfbd795  jiva-ctlg   run      running  11/25/16 09:20:04 UTC


vagrant@client-01:~$ nomad node-status
ID        DC   Name       Class   Drain  Status
dbfbd795  dc1  client-02  <none>  false  ready
b141e876  dc1  client-02  <none>  false  down
dc3199f3  dc1  client-01  <none>  false  ready


amit:demo$ vagrant status
Current machine states:

server-01                 running (virtualbox)
client-01                 running (virtualbox)
client-02                 running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
amit:demo$ vagrant destroy client-02
    client-02: Are you sure you want to destroy the 'client-02' VM? [y/N] y
==> client-02: Forcing shutdown of VM...
==> client-02: Destroying VM and associated drives...


vagrant@client-01:~$ nomad status ctl
ID          = ctl-reps-jiva-vol
Name        = ctl-reps-jiva-vol
Type        = service
Priority    = 50
Datacenters = dc1
Status      = running
Periodic    = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
jiva-ctlg   0       0         1        0       0         1
jiva-repg   1       0         1        0       0         1

Placement Failure
Task Group "jiva-repg":
  * Constraint "distinct_hosts" filtered 1 nodes

Allocations
ID        Eval ID   Node ID   Task Group  Desired  Status   Created At
d87b2c49  2175598a  dc3199f3  jiva-ctlg   run      running  11/25/16 09:30:51 UTC
21b74737  be3c7310  dbfbd795  jiva-repg   stop     lost     11/25/16 09:20:04 UTC
803073c1  be3c7310  dc3199f3  jiva-repg   run      running  11/25/16 09:20:04 UTC
b109dd93  be3c7310  dbfbd795  jiva-ctlg   stop     lost     11/25/16 09:20:04 UTC


amit:demo$ vagrant up client-02
Bringing machine 'client-02' up with 'virtualbox' provider...
==> client-02: Importing base box 'cbednarski/ubuntu-1404'...
==> client-02: Matching MAC address for NAT networking...


vagrant@client-02:~$ nomad node-status
ID        DC   Name       Class   Drain  Status
22cfa519  dc1  client-02  <none>  false  ready
dbfbd795  dc1  client-02  <none>  false  down
b141e876  dc1  client-02  <none>  false  down
dc3199f3  dc1  client-01  <none>  false  ready


vagrant@client-02:~$ nomad status ctl
ID          = ctl-reps-jiva-vol
Name        = ctl-reps-jiva-vol
Type        = service
Priority    = 50
Datacenters = dc1
Status      = running
Periodic    = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
jiva-ctlg   0       0         1        0       0         1
jiva-repg   0       0         2        0       0         1

Allocations
ID        Eval ID   Node ID   Task Group  Desired  Status   Created At
17262ec7  05752784  22cfa519  jiva-repg   run      running  11/25/16 09:39:19 UTC
d87b2c49  2175598a  dc3199f3  jiva-ctlg   run      running  11/25/16 09:30:51 UTC
21b74737  be3c7310  dbfbd795  jiva-repg   stop     lost     11/25/16 09:20:04 UTC
803073c1  be3c7310  dc3199f3  jiva-repg   run      running  11/25/16 09:20:04 UTC
b109dd93  be3c7310  dbfbd795  jiva-ctlg   stop     lost     11/25/16 09:20:04 UTC
vagrant@client-02:~$ 

The specs snippet that has made above possible is mentioned below:
NOTE - This specs snippet belong to the replica group only

group "jiva-repg" {
	
	  count = 2
	  
	  constraint {
                operator  = "distinct_hosts"
          }
		
          restart {			
	        mode = "fail"
	  }

# ...
}