New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Resource for managing AWS Glue Crawlers #3875

Closed
bflad opened this Issue Mar 22, 2018 · 17 comments

Comments

Projects
None yet
4 participants
@bflad
Contributor

bflad commented Mar 22, 2018

Split feature request from #1416. Please note there are relevant comments in #1416 about this feature request. For issues not immediately being worked on, please use 馃憤 upvotes on this original issue comment to help guage community interest.

Terraform Version

terraform 0.10+
terraform-provider-aws (new feature)

Affected Resource(s)

  • aws_glue_crawler

Terraform Configuration Files

# New Resource - not implemented
resource "aws_glue_crawler" "example" {
  # ...
}

Expected Behavior

Create, read, update, and delete support for AWS Glue crawlers.

Actual Behavior

Resource not implemented.

References

@bflad

This comment has been minimized.

Contributor

bflad commented Mar 22, 2018

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented Apr 4, 2018

Thanks @bflad

Are you able to help further with the issue I was having a few weeks ago?

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented Apr 16, 2018

@bflad Are you able to help with the failing test I have?

Context:

I have written the code to support a basic Glue Crawler with the required fields set. The acceptance test is failing with permissions issues. It complains that the service role for Glue cannot assume a role.

If you could help me unblock this then I'll be able to continue. I'm stumped!

@cloudyparts

This comment has been minimized.

cloudyparts commented Apr 27, 2018

@darrenhaken are you still blocked? Do you have a fork somewhere?

@cloudyparts

This comment has been minimized.

cloudyparts commented Apr 27, 2018

@darrenhaken - Found your fork.

I think you might have the wrong ARN for the AWSGlueServiceRole on line 167.

I think the ARN is arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole.

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented Apr 28, 2018

@cloudyparts I tried changing the role as you suggested but I still get:

--- FAIL: TestAccAWSGlueCrawler_basic (20.37s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

		* aws_glue_catalog_crawler.test: 1 error(s) occurred:

		* aws_glue_catalog_crawler.test: error creating Glue crawler: InvalidInputException: Service is unable to assume role arn:aws:iam::697329683179:role/AWSGlueServiceRoleDefault. Please verify role's TrustPolicy
			status code: 400, request id: a4f55ff2-4b19-11e8-8491-a9b7762b818e

FYI I've pushed to my fork the latest changes I have done, it's on branch glue_crawlers

@cloudyparts

This comment has been minimized.

cloudyparts commented Apr 28, 2018

@darrenhaken I don't see anything apparent that would cause an 'assume role' error.

That said I have had better luck with the data.aws_iam_policy_document resource as opposed to inline policies.

You could try this => https://gist.github.com/cloudyparts/f7a80e940cf9648eb815bca6a0d43788

Note: I created this gist with code from my own modules which has no issues. (terraform=v0.11.7, terraform-provider-aws=1.16.0)

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented Apr 29, 2018

@cloudyparts I tried your gist but it still failed, I wonder if there's an issue on the account. Thoughts?
I had set up an empty sandbox account to do testing on this.

@cloudyparts

This comment has been minimized.

cloudyparts commented May 1, 2018

@darrenhaken it is difficult to understand the extent of the issue. I would try creating a role manually in the console then referencing the role arn in your test. See if you can get the crawler created... - I am still puzzled by the error output...

Nothing should be assuming the Glue role except for the glue service when the crawler is invoked. You are only creating the crawler ... not invoking it.

@bflad

This comment has been minimized.

Contributor

bflad commented May 1, 2018

IAM is an eventually consistent service so creating new roles/policies/attachments might take a minute or two to propagate through all AWS services. These tend to bubble up as seemingly fatal errors like "role not found" or "unable to assume role" initially, but will work after some time.

For a large portion of other services we tend to wrap these errors in a resource.Retry() loop for a minute. Here is some example code:

err := resource.Retry(1 * time.Minute, func() *resource.RetryError {
  _, err := conn.Example(input) // current call that is validating IAM parameters and failing
  if err != nil {
    if isAWSErr(err, "InvalidInputException", "Service is unable to assume role") {
      return resource.RetryableError(err)
    }
    return resource.NonRetryableError(err)
  }
  return nil
})
if err != nil {
  return fmt.Errorf("error doing XXX: %s", err)
}
@bflad

This comment has been minimized.

Contributor

bflad commented May 8, 2018

@darrenhaken do you need any more help? Are you able to submit a pull request with what you have so far? We would love to get this feature request into the project.

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented May 8, 2018

@bflad yeah I'm still having trouble getting this working around the assume role. I thought I'd already opened a PR, I'll do that now.

I'm having to let others contribute too of course :)

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented May 8, 2018

See PR for WIP #4484

@bflad bflad added this to the v1.24.0 milestone Jun 20, 2018

@bflad

This comment has been minimized.

Contributor

bflad commented Jun 20, 2018

Kudos to @darrenhaken -- the new aws_glue_crawler resource is merged into master and will release later today. 馃帀

@bflad bflad closed this Jun 20, 2018

@bflad

This comment has been minimized.

Contributor

bflad commented Jun 25, 2018

This has been released in version 1.24.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@darrenhaken

This comment has been minimized.

Contributor

darrenhaken commented Jun 27, 2018

@bflad thanks for the updates! Seems to all be working OK :)

@rehevkor5

This comment has been minimized.

Contributor

rehevkor5 commented Oct 26, 2018

I was experiencing the same error as @darrenhaken ("InvalidInputException: Service is unable to assume role arn:aws:iam::00000000:role/AWSGlueServiceRole-foo. Please verify role's TrustPolicy"), and eventually I realized that this was because I was following the example in the documentation which recommends that you use the ".name" property of the IAM Role in order to populate the ".role" property of the aws_glue_crawler. However, I happened to use a non-default "path" on my IAM Role. Unfortunately, "aws_iam_role.name" does not include the path. It started working when I switched to referring to the Role by ARN instead.

I made this improvement to the docs as a result: #6285

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment