DigitalOcean: Error creating tags

[hlummis]

OS / Environment

MacOS

Ansible version

2.2.0.0

Version of components from requirements.txt

msrestazure: 0.4.17
setuptools: 38.2.3
ansible: 2.2.0.0
dopy: 0.3.5
boto: 2.48.0
boto3: 1.4.8
azure: 2.0.0rc5
msrest: 0.4.1
apache-libcloud: 2.2.1
six: 1.11.0
pyopenssl: 17.4.0
jinja2: 2.8

Summary of the problem

Digitalocean deployment fails when setting tags

Steps to reproduce the behavior

Run ./algo for digitalocean region #8, all [y/N] options set to yes.

The way of deployment (cloud or local)

local

Expected behavior

Ansible sets the tags for the droplet

Actual behavior

Get an error:
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Failed as requested from task"}

Full log

What provider would you like to use?
1. DigitalOcean
2. Amazon EC2
3. Microsoft Azure
4. Google Compute Engine
5. Install to existing Ubuntu 16.04 server

Enter the number of your desired provider
: 1

Enter your API token. The token must have read and write permissions (https://cloud.digitalocean.com/settings/api/tokens):
[pasted values will not be displayed]
:

Name the vpn server:
[algo.local]:

What region should the server be located in?
1. Amsterdam (Datacenter 2)
2. Amsterdam (Datacenter 3)
3. Frankfurt
4. London
5. New York (Datacenter 1)
6. New York (Datacenter 2)
7. New York (Datacenter 3)
8. San Francisco (Datacenter 1)
9. San Francisco (Datacenter 2)
10. Singapore
11. Toronto
12. Bangalore
Enter the number of your desired region:
[7]: 8

Do you want macOS/iOS clients to enable "VPN On Demand" when connected to cellular networks?
[y/N]: y

Do you want macOS/iOS clients to enable "VPN On Demand" when connected to Wi-Fi?
[y/N]: y

List the names of trusted Wi-Fi networks (if any) that macOS/iOS clients exclude from using the VPN (e.g., your home network. Comma-separated value, e.g., HomeNet,OfficeWifi,AlgoWiFi)
:

Do you want to install a DNS resolver on this VPN server, to block ads while surfing?
[y/N]: y

Do you want each user to have their own account for SSH tunneling?
[y/N]: y

Do you want to apply operating system security enhancements on the server? (warning: replaces your sshd_config)
[y/N]: y

Do you want the VPN to support Windows 10 or Linux Desktop clients? (enables compatible ciphers and key exchange, less secure)
[y/N]: y

Do you want to retain the CA key? (required to add users in the future, but less secure)
[y/N]: y

PLAY [Configure the server] ****************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Generate the SSH private key] ********************************************
ok: [localhost]

TASK [Generate the SSH public key] *********************************************
ok: [localhost]

TASK [Change mode for the SSH private key] *************************************
ok: [localhost]

TASK [Ensure the dynamic inventory exists] *************************************
ok: [localhost]

TASK [cloud-digitalocean : Set the DigitalOcean Access Token fact] *************
ok: [localhost]

TASK [cloud-digitalocean : Delete the existing Algo SSH keys] ******************
FAILED - RETRYING: TASK: cloud-digitalocean : Delete the existing Algo SSH keys (10 retries left).
ok: [localhost]

TASK [cloud-digitalocean : Upload the SSH key] *********************************
changed: [localhost]

TASK [cloud-digitalocean : Creating a droplet...] ******************************
ok: [localhost]

TASK [cloud-digitalocean : Add the droplet to an inventory group] **************
changed: [localhost]

TASK [cloud-digitalocean : set_fact] *******************************************
ok: [localhost]

TASK [cloud-digitalocean : Tag the droplet] ************************************
failed: [localhost] (item=staging) => {"failed": true, "item": "staging", "msg": "error tagging resource '73204383': param is missing or the value is empty: resources"}
failed: [localhost] (item=dbserver) => {"failed": true, "item": "dbserver", "msg": "error tagging resource '73204383': param is missing or the value is empty: resources"}
Debugger invoked
(debug) c

TASK [cloud-digitalocean : debug] **********************************************
ok: [localhost] => {
"fail_hint": [
"Sorry, but something went wrong!",
"Please check the troubleshooting guide.",
"https://trailofbits.github.io/algo/troubleshooting.html"
]
}

TASK [cloud-digitalocean : fail] ***********************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Failed as requested from task"}
Debugger invoked
(debug) c

PLAY RECAP *********************************************************************
localhost : ok=12 changed=2 unreachable=0 failed=1

[mackwage]

Am also experience this issue.

[thalessr]

I added the droplet's tag by hand. The tag is Environment:Algo and after that, the installation worked

[Mottoweb]

same problem

[gtb104]

@thalessr, how/where did you add the tag?

[thalessr]

@gtb104, basically follow this tutorial https://www.digitalocean.com/community/tutorials/how-to-tag-digitalocean-droplets and as a the droplet's tag, add Environment:Algo because the installation's script will find the droplet by this tag.
A reminder in the command ansible-playbook deploy.yml -t digitalocean,vpn,cloud -e 'do_access_token=my_secret_token do_server_name=algo.local do_region=ams2' specifies algo.local as your droplet name and the tag must be add on this droplet.

[gtb104]

@thalessr, after tagging, it there anything you need to do to "continue the installation", or was that the last step?

[dguido]

If there's a problem with the readme, please submit a PR!

[davidemyers]

Looks like an issue has been opened against Ansible.

[thalessr]

@gtb104 yes, I forgot to mention it.. my bad =(. However, it is needed to rerun the installation commands with the same parameters. Eg: if in your first attempt it was used ansible-playbook deploy.yml -t digitalocean,vpn,cloud -e 'do_access_token=my_secret_token do_server_name=algo.local do_region=ams2' rerun it after adding the tags.

[Ducki]

Until the patch has been merged and ansible gets updated, as a quick workaround you can make the little change from the commit in your local installation (see changed code in the link).

[davidemyers]

Here's a patch file you can use to get going again. Save the following patch to your algo directory in a file called, for example, do_tag.patch. Then run patch -b -p0 < do_tag.patch.

--- ./env/lib/python2.7/site-packages/ansible/modules/core/cloud/digital_ocean/digital_ocean_tag.py.orig	2017-12-02 09:42:56.044524930 -0500
+++ ./env/lib/python2.7/site-packages/ansible/modules/core/cloud/digital_ocean/digital_ocean_tag.py	2017-12-02 09:43:14.245074861 -0500
@@ -242,7 +242,7 @@
     module = AnsibleModule(
         argument_spec=dict(
             name=dict(type='str', required=True),
-            resource_id=dict(aliases=['droplet_id'], type='int'),
+            resource_id=dict(aliases=['droplet_id'], type='str'),
             resource_type=dict(choices=['droplet'], default='droplet'),
             state=dict(choices=['present', 'absent'], default='present'),
             api_token=dict(aliases=['API_TOKEN'], no_log=True),

[tscizzle]

@thalessr Hmm, I added that tag (Environment:Algo ) to my droplet in Digital Ocean, but re-running the deploy script yields the same error as before. @gtb104 Were you able to get it to work this way? Was there anything else you had to do to make it work?

[useradd-deploy]

Thanks to @davidemyers, I created his patch file and ran the patch command before running ./algo. The Algo script completed successfully.

[tscizzle]

Confirmed that worked for me. Thank you!

[Mottoweb]

@davidemyers works like a charm, thanks a lot!

[ghost]

The patch unfortunately does not work for me :(

TASK [cloud-digitalocean : Tag the droplet] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "error tagging resource '73812578': The resource you were accessing could not be found."}

Do you guys have any other workaround? Thanks!

[useradd-deploy]

1. @beachfork Perhaps try another cloud server provider.

2. @dguido @gunph1ld FYI, based upon two now closed Anisible issues 33459 and 33486, it appears that the error is caused because Digital Ocean changed its API to treat the tag argument as a string instead of a number.

[jackivanov]

Can't reproduce now. Looks like DO accepts both int and str now. Could anyone verify also?

[davidemyers]

I agree, with the patch I posted above backed out I can create droplets again.

[ElliotSanchez]

@gunph1ld This has started to fail again without the patch, and resolves with the patch applied.

[spartanhooah]

Still fails with the patch.

[TACIXAT]

Confirmed failing with and without the patch.

[TACIXAT]

Patch works, I had to actually use the patch (I was doing it by hand in a copied directory, something was failing in that workflow).

Confirmed with their API in CURL that integers are now not accepted. A quoted version did successfully tag my droplet.

$ curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $KEY" -d '{"resources":[{"resource_id":$RESOURCE,"resource_type":"droplet"}]}' "https://api.digitalocean.com/v2/tags/awesome/resources"
{"id":"parameter_missing","message":"param is missing or the value is empty: resources"}

[TACIXAT]

The fix looks live in Ansible, maybe it hasn't reached pip yet. Until that hits, just patch manually.

[jmerizia]

Alternatively, to solve this temporarily, just upgrade the the devel branch of Ansible. Just run the following:

python -m pip --upgrade install git+https://github.com/ansible/ansible@devel

This will install the bleeding edge Ansible version 2.5.0.
It is generally a bad idea to do this, as certain features could potentially be deprecated in newer versions, but I can verify that I only get deprecation warnings, and that deploying works properly. I haven't thoroughly tested hosting providers other than DigitalOcean however.
But like @douggard said, it should be soon that the fix reaches pip anyways.

[GavinSchalliol]

I've tried both applying the patch and upgrading to the Ansible devel branch. I still see an error on the same step but the error has changed.

Before patching/upgrading:

TASK [cloud-digitalocean : Tag the droplet] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "error tagging resource '78855946': param is missing or the value is empty: resources"}

After patching/upgrading:

TASK [cloud-digitalocean : Tag the droplet] ***************************************************************************************************
fatal: [localhost]: FAILED! => {"attempts": 1, "changed": false, "msg": "error tagging resource '78855946': The resource you were accessing could not be found."}

It works if I manually tag the droplet as environment:algo and then try again.

[spartanhooah]

Even manually tagging the droplet after patching didn't work for me. I get the first message @GavinSchalliol posted. I have verified that the patch has been applied to digital_ocean_tag.py

[spartanhooah]

Ok. Same thing (I removed the creatures to make the logs more compact):

_____________________________
< PLAY [Configure the server] >
-----------------------------
______________
< TASK [setup] >
--------------
ok: [localhost]
_____________________________________
< TASK [Generate the SSH private key] >
-------------------------------------
ok: [localhost]
____________________________________
< TASK [Generate the SSH public key] >
------------------------------------
ok: [localhost]
____________________________________________
< TASK [Change mode for the SSH private key] >
--------------------------------------------
ok: [localhost]
____________________________________________
< TASK [Ensure the dynamic inventory exists] >
--------------------------------------------
ok: [localhost]
________________________________________________________
/ TASK [cloud-digitalocean : Set the DigitalOcean Access \
\ Token fact]                                            /
--------------------------------------------------------
ok: [localhost]
_________________________________________________________
/ TASK [cloud-digitalocean : Delete the existing Algo SSH \
\ keys]                                                   /
---------------------------------------------------------
FAILED - RETRYING: TASK: cloud-digitalocean : Delete the existing Algo SSH keys (10 retries left).
ok: [localhost]
________________________________________________
< TASK [cloud-digitalocean : Upload the SSH key] >
------------------------------------------------
changed: [localhost]
___________________________________________________
< TASK [cloud-digitalocean : Creating a droplet...] >
---------------------------------------------------
ok: [localhost]
____________________________________________________________
/ TASK [cloud-digitalocean : Add the droplet to an inventory \
\ group]                                                     /
------------------------------------------------------------
changed: [localhost]
______________________________________
< TASK [cloud-digitalocean : set_fact] >
--------------------------------------
ok: [localhost]
_____________________________________________
< TASK [cloud-digitalocean : Tag the droplet] >
---------------------------------------------
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "error tagging resource '82815151': The resource you were accessing could not be found."}
___________________________________
< TASK [cloud-digitalocean : debug] >
-----------------------------------
ok: [localhost] => {
   "fail_hint": [
       "Sorry, but something went wrong!", 
       "Please check the troubleshooting guide.", 
       "https://trailofbits.github.io/algo/troubleshooting.html"
   ]
}
__________________________________
< TASK [cloud-digitalocean : fail] >
----------------------------------
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Failed as requested from task"}
____________
< PLAY RECAP >
------------
localhost                  : ok=12   changed=2    unreachable=0    failed=1   

[ghost]

I'm having the same error output as spartanhooah, I'm using a brand new DO droplet running Ubuntu 16 as the client machine, I've tried the ansible2.4 branch and still no success. Maybe a workaround would be to remove the whole tagging process?

[GeYebo]

I see the same problem as @beachfork and @spartanhooah. I downloaded a fresh version today. If I try to add a tag to the droplet and then run algo I get the following error:
TASK [Check the system] ********************************************************
fatal: [???.65.225.???]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '???.65.225.???' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey).\r\n", "unreachable": true}

[spartanhooah]

Is this issue still being investigated? Is there anything more the handful of us having the problem can do to help?

[GeYebo]

I second @spartanhooah's question.

[dguido]

I can't reproduce this issue either and I primarily use Algo on DigitalOcean. As Jack said, please delete and redownload Algo. You may also want to clean up your Digital Ocean account (ssh keys, etc).

[spartanhooah]

I'll try cleaning up the DO account, but I've already started from scratch on my local machine several times.

[ghost]

@spartanhooah I've finally figured out what was going on, it's not on Algo side, but it's something wrong on DO side. I've used doctl to manually clear my tags, using the command doctl compute tag del X and managed to remotely deploy algo normally! Hope it helps!

[jproz]

@beachfork where did you put this new command? does the "X" represent a tag or is that part of the command? I'm not a DO expert.

[jproz]

I also sent a -vvvv log to @dguido via slack

[ghost]

@jproz Hey! Use doctl compute tag ls to see all the tags the you have, but you just need to clear the environment:algo one, so it would be doctl compute tag del enviroment:algo. Try to use the help with the doctl command when in doubt.

[GeYebo]

@beachfork So is it a problem when there is already an existing tag? For instance, I already have an algo vpn server running and I am looking to build a new one before tearing down the old one. Thanks.

[GeYebo]

@dguido I tried @beachfork's suggestion; I deleted the tag on the existing vpn server; I put new ssh keys on DO. I continue to get the same error. What else can I try?

[jproz]

OK Guys - so I finally got around to doing what @beachfork suggested and everything worked (finally - after weeks and weeks). @dguido @jackivanov - It appears the issues is somewhere around tag reuse (specifically environment:algo) - I had to do a on doctl tag delete --force to get it to work.

[dguido]

That’s excellent to hear. Jack and I are still unable to reproduce this issue. Can you submit a PR for the troubleshooting guide to address it? Thanks!

[spartanhooah]

@beachfork @kenydub how do I run that command? @beachfork mentioned snap in the gitter chat, but I've never heard of it.

[GeYebo]

@jproz @dguido It works!
@spartanhooah Here is what I did:

1. Download doctl.
2. Run doctl auth init; it will ask you for your token which you can get (or generate) on the API tab at DigitalOcean.
3. Once you are authorized on DO, you can run doctl compute tag list to see the list of tags;
4. Run doctl compute tag delete enivronment:algo --force to delete the environment:algo tag.
5. Finally run doctl compute tag list to make sure that the tag has been deleted.
6. Run algo as directed.

Very strange error indeed. Truly a case of bit rot...

[spartanhooah]

That did it, thank you!!

[GeYebo]

@dguido I don't know if this is a useful observation for you or not, but I noticed that the tag on DO is now Environment:Algo whereas it used to be environment:algo. Is it possible they changed the system to recognize case. If @jproz or @beachfork have not submitted a PR for the trouble shooting guide, I will do it.

[y0ast]

@kenydub There's a spelling mistake in your proposed command 4.:

This is correct:
doctl compute tag delete environment:algo --force

(not enivronment)

This is a great fix btw :)

[pguizeline]

The fix for this would probably be something like - during the install if the user picks DO as the provider, the Algo scripts would use the user given API key and send a curl command with the following:

curl -X DELETE -H "Content-Type: application/json" -H "Authorization: Bearer USER_API_KEY" "https://api.digitalocean.com/v2/tags/environment:algo"

I'm sorry I can't be of any more help since I'm completely lost with Ansible, and I don't know if the maintainers would consider this a dirty fix, since trying to delete a tag that actually maybe not even exist. This problem seems to affect users that have been using Algo from a long time, before DO changed something with the tagging on their side.