Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem retrieving custom EC2 tags | HTTPError: HTTP Error 404: Not Found #1884

Closed
nodesocket opened this issue Sep 2, 2015 · 8 comments
Closed

Problem retrieving custom EC2 tags | HTTPError: HTTP Error 404: Not Found #1884

nodesocket opened this issue Sep 2, 2015 · 8 comments
Assignees
@yannmh
Milestone
5.7.0

Comments

@nodesocket
Copy link

Seeing the following logged for dd-collector using the AWS integration.

2015-09-02 18:49:56 UTC | ERROR | dd.collector | util(util.py:348) | Problem retrieving custom EC2 tags
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/util.py", line 336, in get_tags
    iam_role = urllib2.urlopen(EC2.METADATA_URL_BASE + "/iam/security-credentials").read().strip()
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
@mweagle
Copy link

mweagle commented Sep 3, 2015

Hi @nodesocket - I submitted a PR in that area to support multiple AWS regions. I will resubmit an updated one and would like to address this issue. Can you provide a bit more information about your EC2 state?

  1. Are you associating IAM Roles with your EC2 instance(s)?
  2. What is the output of curl http://169.254.169.254/latest/meta-data/iam/info
  3. What is the output of curl http://169.254.169.254/latest/meta-data/iam/info/ (The trailing slash is important)
  4. What is the output of curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

Thanks.

@nodesocket
Copy link
Author

1.) Yes, associating IAM roles on some instances, but the instances am seeing the error from no.

2.) On the instances I am seeing the error from:

ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/info
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

3.)

ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/info/
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

4.)

ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

@nodesocket
Copy link
Author

Any update on this issue? Still seeing these errors.

Sep 15 16:46:10 processing1 dd.collector: ERROR (util.py:348): Problem retrieving custom EC2 tags#012Traceback (most recent call last):#12 File "/opt/datadog-agent/agent/util.py", line 343, in get_tags#012 tag_object = connection.get_all_tags({'resource-id': EC2.metadata['instance-id']})#12 File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/boto/ec2/connection.py", line 4177, in get_all_tags#012 [('item', Tag)], verb='POST')#12 File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/boto/connection.py", line 1185, in get_list#012 raise self.ResponseError(response.status, response.reason, body)#012EC2ResponseError: EC2ResponseError: 403 Forbidden#012#12UnauthorizedOperationYou are not authorized to perform this operation.9bf53ee1-74d6-4fb2-84d4-0c04f88cd18d

@yannmh
Copy link
Member

yannmh commented Sep 16, 2015

Sorry for the late answer @nodesocket.

Datadog agent relies on (among others) http://169.254.169.254/latest/meta-data/iam/info endpoint to retrieve the instance's metadata. If the endpoint is unavailable or if it returns an HTTP error status code, then no EC2 tag is fetched.

We've been running some tests, and found that an IAM role assignation is required to be able to use this endpoint. The error message, I reckon, is quite confusing here: we will likely catch this error to display a more comprehensive one in a next version.

Is this error disrupting the proper functionning of Datadog agent on your instance ?

Thank you

@nodesocket
Copy link
Author

Thanks for the reply. No the error is not disrupting anything, just noticed it in our logs. Perhaps if the instance does not have a IAM role assigned just return instead of showing the 404 error.

@yannmh yannmh added this to the 5.6.0 milestone Sep 17, 2015
@yannmh
Copy link
Member

yannmh commented Sep 17, 2015

Thanks for the further details @nodesocket.
I am assigning the ticket to our 5.6.0 agent milestone, so we'll make sure to handle this case more gracefully 🎨

@remh remh modified the milestones: 5.7.0, 5.6.0 Nov 3, 2015
@yannmh yannmh self-assigned this Dec 28, 2015
yannmh added a commit that referenced this issue Feb 23, 2016
@yannmh
[core] EC2 tags collection requires an IAM role
d2f1287 
An IAM role associated with the instance is required to retrieve AWS EC2
custom tags. Log a warning when missing, not an exception.

Fix #1884
yannmh added a commit that referenced this issue Feb 23, 2016
@yannmh
[core] EC2 tags collection requires an IAM role
0849d6f 
An IAM role associated with the instance is required to retrieve AWS EC2
custom tags. Log a warning when missing, not an exception.

Fix #1884
@yannmh yannmh mentioned this issue Feb 23, 2016
Merged
@nodesocket
Copy link
Author

👍 thanks. When will this ship?

@yannmh
Copy link
Member

yannmh commented Feb 24, 2016

This will be shipped with Datadog Agent 5.7.0 which should be out within two weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yannmh yannmh

Labels
None yet
Projects
None yet
Milestone
5.7.0
Development

No branches or pull requests
4 participants
@mweagle @nodesocket @remh @yannmh