Bugfix/brbdcf 1172 cannot get lock hosts pool deployment

[laurentganne]

Pull Request description

Description of the change

Implemented a workaround for a Consul issue regarding the time to wait for a lock, issue hashicorp/consul#4003
Added a unit test where 50 threads attempt to allocate one host in a Pool of 50 free hosts (test failing on error getting the lock before the workaround).
Increased the max timeout used in Yorc to manage more than 150 simultaneous requests.
For now the lock timeout is not accessible for change to the user, it may be needed depending on the size of deployments we expect to support with Hosts Pools.

Misc change: while doing a test with big hosts pool, noticed a limitation in Consul regarding the number of operations that can be performed in a transaction (64).
Had to split the apply transaction if it contains more operations than this limit in Consul implementation.

What I did

The issue hashicorp/consul#4003 created on Consul is that when a lock is taken with the property LockTryOnce=true, a call to lock.Lock() will attempt to get the lock for an expected duration of LockWaitTime before returning without having succeeded.
Actually with a LockWaitTime of 45 seconds, the unit test with 50 threads failed in around 5 seconds.

Modified our code to not use LockTryOnce=true, and instead as a workaround, arm a timer for a LockWaitTime duration, to close a channel stopChannel after this duration.
This channel stopChannel is passed in argument to lock.Lock() that is blocking until it will detect that stopChannel has been closed.

How to verify it

Initially a deployment of elk-basic with a pool of 4 hosts was reported to fail. Trying again a 4 hosts deployment in development environment.
As a yorc user, generate private/public keys in a directory, check the private key is read-only for your yorc user, add the public key content to a file authorized_keys,
Create a devenv.sh file containing your proxy settings:

HTTP_PROXY=http://1.2.3.4:8080
HTTPS_PROXY=http://1.2.3.4:8080

Then create 4 ssh servers :

docker run -p 8701:22 \
 -e "AUTH_KEY=$(cat $HOME/yorc/testhost_secrets/authorized_keys)" \
 --env-file devenv.sh \
 --rm -d --name host1 --hostname host1 laurentg/ystiahost

docker run -p 8702:22 \
 -e "AUTH_KEY=$(cat $HOME/yorc/testhost_secrets/authorized_keys)" \
 --env-file devenv.sh \
 --rm -d --name host2 --hostname host2 laurentg/ystiahost

docker run -p 8703:22 \
 -e "AUTH_KEY=$(cat $HOME/yorc/testhost_secrets/authorized_keys)" \
 --env-file devenv.sh \
 --rm -d --name host3 --hostname host3 laurentg/ystiahost
 
docker run -p 8704:22 \
 -e "AUTH_KEY=$(cat $HOME/yorc/testhost_secrets/authorized_keys)" \
 --env-file devenv.sh \
 --rm -d --name host4 --hostname host4 laurentg/ystiahost

Verify that as yorc user, you can connect to each host using the private key you generated :

ssh -i <path to secrets>/id_rsa test@myhost -p 8701
ssh -i <path to secrets>/id_rsa test@myhost -p 8702
ssh -i <path to secrets>/id_rsa test@myhost-p 8703
ssh -i <path to secrets>/id_rsa test@myhost-p 8703

Create a Hosts Pool made of these 4 hosts, using a yaml file like https://github.com/laurentganne/ystia-devenv/blob/master/testlab/pool4hosts.yaml
and running :

yorc hp apply pool4hosts.yaml

On a setup where the Yorc plugin was added to Alien4Cloud, Yorc was configured as an orchestrator, and a Hosts Pool location was configured.
From Alien4Cloud, Menu Administration > Orchestrators, select Yorc.
Then in the left-hand-side menu, select Location, On demand resources, and add the following resource : yorc.nodes.hostspool.Compute. No need to set credentials or any other property.

From Alien4Cloud, Menu Catalog > Manage Archives > Git import
add the following Git location:

• Repository URL: https://github.com/alien4cloud/csar-public-library.git
• Branch: develop
• Archive Folder: org/ystia

From the Menu Applications
Create a new Application, Initializing the topology from Topology Template welcome_basic.
Select the Environment, Topology, Click on Edit to edit the topology.
Remove the Component Network.
Then add 3 new Components of type Compute.
And within each of these 3 new Compute components add a component Welcome.

So finally we have 4 compute nodes, with an application Welcome deployed on each.
Click on Save to save changes.
Then go back to Environment. At the Location step, select the Hosts Pool.
Then deploy the application.

When the deployement has started on Yorc, run this command to check the 4 hosts in the pool were allocated for this application :

yorc hp list

Wait until the deployment ends successfuly. Then undeploy the application, wait until it is undeployed successfuly.
Run again the command yorc hp list to check the 4 hosts are now free in the Hosts Pool.

[codecov]

Codecov Report

Merging #15 into develop will increase coverage by 0.07%.
The diff coverage is 70.58%.

@@            Coverage Diff             @@
##           develop     #15      +/-   ##
==========================================
+ Coverage    46.13%   46.2%   +0.07%     
==========================================
  Files           88      88              
  Lines         8256    8267      +11     
==========================================
+ Hits          3809    3820      +11     
  Misses        3820    3820              
  Partials       627     627

Impacted Files	Coverage Δ
prov/hostspool/hostspool_mgr.go	67.8% <70.58%> (+0.51%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4178c4f...5fd738b. Read the comment docs.

[loicalbertin]

LGTM