Tenant loop #77

nabilschear · 2019-03-11T15:12:52Z

change to tenant client api interface

add to an existing node just prints an error
added new update command that will delete and then re-add the node with
new parameters
delete will now confirm that the node was deleted before returning

fixes #75

add to an existing node just prints an error added new udpate command that will delete and then re-add the node with new paramaters delete will now confirm that the node was deleted before returning fixes keylime#45

nabilschear · 2019-03-11T15:15:26Z

@lukehinds i don't seem to be able to tag this PR or assign it. Also this is my first foray into doing this the proper git/contributor way. go easy on me =) . i already managed to reference the wrong issue and make a bunch of typos. i really hate these new mac keyboards

lukehinds · 2019-03-11T15:18:53Z

No worries, I think the issue was that you commit message had four spaces in front of each sentence, which resulted in it being formatted as code (you can see where I removed the 4 spaces from 'fixes').

I just need to work out how this has side stepped travis :)

nabilschear · 2019-03-11T15:21:35Z

Ha! the reason for the spaces is that i've tried probably 5 different ways of getting this pushed. I kept copying my commit message from git log (which puts in the spaces). I tried doing it directly to the keylime repo. made my own fork. then fought with ssh and https auth. finally i found that my local repo was corrupt. I had to go and re-pull it and start all over.

lukehinds · 2019-03-11T16:30:24Z

Seeing a few things wrong (unless I am mistaken)...

I see the node register:

1552321304.57 - keylime.registrar_client - INFO - Registration activated for node D432FBB3-D2F1-4A97-9EF7-75BD81C00000.

I then try to add again:

keylime_tenant -v 127.0.0.1 -t 127.0.0.1 -f somefile --uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000
Using config file /etc/keylime.conf
1552321385.39 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password
1552321385.39 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca
1552321385.4 - keylime.tenant - INFO - TPM PCR Mask from policy is 0x408000
1552321385.4 - keylime.tenant - INFO - vTPM PCR Mask from policy is 0x808000
1552321385.4 - keylime.tenant - DEBUG - b64_v:l4rztbOhPDgS6fkBwIvW9C/1RpZP4muNEw7JnlQpDcY=
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/keylime-1.2-py2.7.egg/keylime/tenant.py", line 745, in main
    mytenant.do_cv()
  File "/usr/lib/python2.7/site-packages/keylime-1.2-py2.7.egg/keylime/tenant.py", line 481, in do_cv
    raise Exception("Node %s already existed at CV.  Please use delete or update."%self.node_uuid)
Exception: Node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 already existed at CV.  Please use delete or update.
1552321385.41 - keylime.tenant - ERROR - None
1552321385.41 - keylime.tenant - ERROR - Error: Node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 already existed at CV.  Please use delete or update.

1552321702.79 - keylime.cloudverifier - WARNING - Node of uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 already exists

Good, that's as expected. A small nit: Is it possible to only output the Exception (last line) and not the python stack trace (or maybe that's because its in DEBUG mode)?

I then try to delete:

keylime_tenant -v 127.0.0.1 -t 127.0.0.1 --uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 -c delete
Using config file /etc/keylime.conf
1552321767.32 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password
1552321767.32 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca
1552321772.39 - keylime.tenant - ERROR - Timed out waiting for delete of node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 to complete at CV

1552321767.34 - keylime.cloudverifier - INFO - DELETE returning 200 response for instance id: D432FBB3-D2F1-4A97-9EF7-75BD81C00000

lukehinds · 2019-03-11T16:48:02Z

keylime/tenant.py

+        deleted = False
+        for _ in range(12):
+            response = tornado_requests.request("GET", "http://%s:%s/instances/%s"%(self.cloudverifier_ip,self.cloudverifier_port,self.node_uuid),context=self.context)
+            if response.status_code == 404:


I think you should be looking for a 200 here?

DELETE returning 200 response for instance id: D432FBB3-D2F1-4A97-9EF7-75BD81C00000

actually that call is to get the status of the node, so i want i to show a 404 to confirm that the node's been deleted from the database. do you see any logs on the verifier that show that the status request is being serviced?

actually the verifier doesn't log 404s i think.

https://github.com/keylime/python-keylime/blob/master/keylime/cloud_verifier_tornado.py#L114

It looks like the delete is actually failing somehow:

[root@localhost ~]# keylime_tenant -v 127.0.0.1 -t 127.0.0.1 --uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 -c delete Using config file /etc/keylime.conf 2019-03-11 17:09:29.664 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password 2019-03-11 17:09:29.664 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca 2019-03-11 17:09:34.739 - keylime.tenant - ERROR - Timed out waiting for delete of node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 to complete at CV

verifier:

2019-03-11 17:09:29.685 - keylime.cloudverifier - INFO - DELETE returning 200 response for instance id: D432FBB3-D2F1-4A97-9EF7-75BD81C00000

I then make up a UUID:

[root@localhost ~]# keylime_tenant -v 127.0.0.1 -t 127.0.0.1 --uuid D432FBB3-D2F1-4A97-9EF7-75BD81CXXXXX -c delete Using config file /etc/keylime.conf 2019-03-11 17:09:45.092 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password 2019-03-11 17:09:45.092 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca 2019-03-11 17:09:45.106 - keylime.tenant - ERROR - Delete command response: 404 Unexpected response from Cloud Verifier. 2019-03-11 17:09:45.106 - keylime.tenant - ERROR - Response code 404: instance id not found

2019-03-11 17:09:45.104 - keylime.cloudverifier - INFO - DELETE returning 404 response. instance id: D432FBB3-D2F1-4A97-9EF7-75BD81CXXXXX not found.

The message "Timed out waiting for delete of node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 to complete at CV" actually comes from the status check. the made up uuid works as expected. can you try doing an add after the the failed delete?

I think we might be able to ignore the above, this is from when a failed tenant quote state exists.

Some more details.

Get the node into a failed quote state:

2019-03-11 17:46:28.144 - keylime.cloudverifier - WARNING - Instance D432FBB3-D2F1-4A97-9EF7-75BD81C00000 has failed tenant quote. stopping polling

Try to delete the node:

keylime_tenant -v 127.0.0.1 -t 127.0.0.1 --uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 -c delete Using config file /etc/keylime.conf 2019-03-11 17:50:00.290 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password 2019-03-11 17:50:00.290 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca 2019-03-11 17:50:05.409 - keylime.tenant - ERROR - Timed out waiting for delete of node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 to complete at CV

If its a registered node not in a failed quote state, it deletes ok:

keylime_tenant -v 127.0.0.1 -t 127.0.0.1 --uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 -c delete Using config file /etc/keylime.conf 2019-03-11 17:48:25.801 - keylime.tenant - WARNING - CAUTION: using default password for private key, please set private_key_pw to a strong password 2019-03-11 17:48:25.801 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca 2019-03-11 17:48:25.844 - keylime.tenant - INFO - Node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 deleted from the CV

The verifier is not giving us much more than the 200 each time:

2019-03-11 17:51:46.945 - keylime.cloudverifier - INFO - DELETE returning 200 response for instance id: D432FBB3-D2F1-4A97-9EF7-75BD81C00000

yeah, the verifier just marks the node for deletion and returns 200. I thought that this code would just immediately delete it if it was already failed
https://github.com/keylime/python-keylime/blob/7b0a5b4c57e0efd7add2cac88a7ad0eca95e9a90/keylime/cloud_verifier_tornado.py#L151-L154

I'll debug a little and see what's going on

hrm, i wasn't able to reproduce this one. when it is in a failed state it does delete for me.

nabilschear · 2019-03-11T18:46:53Z

@lukehinds try this version. behaves differently depending on the state of the node on the verifier

nabilschear · 2019-03-15T14:05:11Z

@lukehinds i wasn't sure if you'd checked this latest branch for your problem when you posted this comment?
#75 (comment)

let me know if you're still having trouble.

lukehinds · 2019-03-15T17:25:11Z

@lukehinds i wasn't sure if you'd checked this latest branch for your problem when you posted this comment?
#75 (comment)

let me know if you're still having trouble.

Thanks buddy, should be testing it shortly. I want to get this one in for the next release too

lukehinds · 2019-03-18T11:45:14Z

hi @nabilschear

this still fails for me

here is the complete session: https://gist.github.com/lukehinds/a624d1672628adbccf5a95a83252fb20

I am testing this out on a fresh install everytime (that passes run_tests.sh first).

Just let me know if you would like me to try anything else.

jetwhiz · 2019-03-18T20:10:51Z

Is this being run on a machine with an emulated TPM? If so, it looks like EK cert checking is not turned off: "Error processing ek/ekcert". This is throwing an exception which might be causing other issues, too.

lukehinds · 2019-03-18T21:52:46Z

Good point, will check first thing in the morning!

lukehinds · 2019-03-19T08:08:51Z

Apologies for wasting a few cycles. I thought my set up script was toggling off require ek, but it seems not.

So this works better for me now, the last point I have is why use raise Exception and not logger.error - when we raise an Error we get the following stack trace outputted, that kind of suggests something is broken, when its not - the code is just reporting the state (it already exists).

1552982503.81 - keylime.cloudverifier - WARNING - Node of uuid D432FBB3-D2F1-4A97-9EF7-75BD81C00000 already exists
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/keylime-1.2-py2.7.egg/keylime/tenant.py", line 744, in main
    mytenant.do_cv()
  File "/usr/lib/python2.7/site-packages/keylime-1.2-py2.7.egg/keylime/tenant.py", line 481, in do_cv
    raise Exception("Node %s already existed at CV.  Please use delete or update."%self.node_uuid)
Exception: Node D432FBB3-D2F1-4A97-9EF7-75BD81C00000 already existed at CV.  Please use delete or update.

I don't think you need to change it in this patch. I will create a new issue and we can discuss removing them all from tenant.py

nabil added 2 commits March 11, 2019 11:10

change to tenant client api interface

c67f278

add to an existing node just prints an error added new udpate command that will delete and then re-add the node with new paramaters delete will now confirm that the node was deleted before returning fixes keylime#45

change to tenant client api interface

39e1ae1

add to an existing node just prints an error added new udpate command that will delete and then re-add the node with new paramaters delete will now confirm that the node was deleted before returning fixes keylime#45

lukehinds self-requested a review March 11, 2019 15:19

don't use a global tpm object for this test code

4ada219

changing stack trace to be ouptut in debug log level

25142cf

lukehinds reviewed Mar 11, 2019

View reviewed changes

return 202 when the node can't be deleted immediately

82b9c67

test on valid node will return 202

04d3afe

lukehinds approved these changes Mar 19, 2019

View reviewed changes

lukehinds merged commit 5a03f96 into keylime:master Mar 19, 2019

nabilschear deleted the tenant_loop branch October 16, 2019 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tenant loop #77

Tenant loop #77

nabilschear commented Mar 11, 2019 •

edited by lukehinds

Loading

nabilschear commented Mar 11, 2019

lukehinds commented Mar 11, 2019

nabilschear commented Mar 11, 2019

lukehinds commented Mar 11, 2019 •

edited

Loading

lukehinds Mar 11, 2019

nabilschear Mar 11, 2019

nabilschear Mar 11, 2019

nabilschear Mar 11, 2019

lukehinds Mar 11, 2019

nabilschear Mar 11, 2019

lukehinds Mar 11, 2019

lukehinds Mar 11, 2019

nabilschear Mar 11, 2019

nabilschear Mar 11, 2019

nabilschear commented Mar 11, 2019

nabilschear commented Mar 15, 2019

lukehinds commented Mar 15, 2019

lukehinds commented Mar 18, 2019

jetwhiz commented Mar 18, 2019 via email

lukehinds commented Mar 18, 2019

lukehinds commented Mar 19, 2019

Tenant loop #77

Tenant loop #77

Conversation

nabilschear commented Mar 11, 2019 • edited by lukehinds Loading

nabilschear commented Mar 11, 2019

lukehinds commented Mar 11, 2019

nabilschear commented Mar 11, 2019

lukehinds commented Mar 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nabilschear commented Mar 11, 2019

nabilschear commented Mar 15, 2019

lukehinds commented Mar 15, 2019

lukehinds commented Mar 18, 2019

jetwhiz commented Mar 18, 2019 via email

lukehinds commented Mar 18, 2019

lukehinds commented Mar 19, 2019

nabilschear commented Mar 11, 2019 •

edited by lukehinds

Loading

lukehinds commented Mar 11, 2019 •

edited

Loading