aws_storagegateway_gateway with gateway_type FILE_S3 400 error

[varunrsystems]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.37.0

Affected Resource(s)

• aws_storagegateway_gateway

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
  required_version = "= 0.13.5"
}

provider "aws" {
  region  = "us-east-2"
  version = "~> 3.0"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.63.0"

  name = "test-vpc"
  cidr = "10.0.0.0/16"

  azs              = ["us-east-2a", "us-east-2b"]
  private_subnets  = ["10.0.0.0/24"]
  public_subnets   = ["10.0.10.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = false
  single_nat_gateway = true

  enable_dns_hostnames               = true
  enable_dns_support                 = true
  enable_dhcp_options = false
}

resource "aws_storagegateway_gateway" "nfs_file_gateway" {
  gateway_ip_address = aws_instance.gateway.public_ip
  gateway_name       = "test"
  gateway_timezone   = "GMT"
  gateway_type       = "FILE_S3"
}

resource "aws_instance" "gateway" {
  ami           = "ami-08fff025e75894630"
  instance_type = "m5.xlarge"

  # Refer to AWS File Gateway documentation for minimum system requirements.
  ebs_optimized = true
  subnet_id     = module.vpc.public_subnets.0

  ebs_block_device {
    device_name           = "/dev/xvdf"
    volume_size           = "150"
    volume_type           = "gp3"
    delete_on_termination = true
  }

  vpc_security_group_ids = [
    aws_security_group.storage_gateway.id,
  ]
}

resource "aws_security_group" "storage_gateway" {
  name        = "test-security-group"
  description = "Allow inbound NFS traffic"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group_rule" "egress_all" {
  description       = "egress"
  from_port         = 0
  protocol          = "ALL"
  security_group_id = aws_security_group.storage_gateway.id
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "ingress_139_tcp_product" {
  description              = "For SMB"
  from_port                = 0
  protocol                 = "all"
  security_group_id        = aws_security_group.storage_gateway.id
  to_port                  = 0
  type              = "ingress"
  cidr_blocks       = ["0.0.0.0/0"]
}

Debug Output

https://gist.github.com/varunrsystems/07076438b639b723c5c70f363d83991a

Panic Output

Expected Behavior

Terraform apply/plan successfully executed

Actual Behavior

Break with error

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "ef88f6ad-a94a-469b-9e81-3e8bfa93830c"
  },
  Error_: {
    ErrorCode: "UnsupportedOperationForGatewayType"
  },
  Message_: "This operation is not valid for the specified gateway."
}

Steps to Reproduce

1. terraform apply

Important Factoids

• It also happen in terraform plan when running for older infrastructure
• Also tried this with aws provider version v3.36.0, v3.35.0, v3.34.0, and facing same issue.
• This issue recently started from 4 days only, before this all things working fine, looks like maybe some AWS api get also changed
• Initially it looks like when gateway_type = FILE_S3 we need to make https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_storagegateway_gateway.go#L572 conditional

References

• #0000

[gdavison]

Thanks for reporting this, @varunrsystems. I've done a bit of research on this, and from what I can tell, you're deploying the storage gateway in us-east-2.

We check for this error at

terraform-provider-aws/aws/resource_aws_storagegateway_gateway.go

Lines 573 to 575 in e1a8d5e

	if err != nil && !isAWSErr(err, storagegateway.ErrCodeInvalidGatewayRequestException, "The specified operation is not supported") {
	return fmt.Errorf("error reading Storage Gateway Bandwidth rate limit: %s", err)
	}

. For some reason, in us-east-2, the AWS API returns a slightly different message than in other regions. Everywhere else, the API returns a message matching The specified operation is not supported, while in us-east-2 the API returns This operation is not valid for the specified gateway

[adamjohnson01]

We are also seeing this in eu-west-1 using 3.37.0

[stekern]

I'm experiencing the same issue in eu-west-1 using AWS provider 3.36.0.

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.

[varunrsystems]

I contacted the AWS support for this change in API response, they forwarded to SGW Service Team to investigate the issue and return with their findings, will update further reply here.

[nerryc]

@gdavison @varunrsystems We are also seeing this in EU-WEST-1 but not in AF-SOUTH-1. Started around 26/27 April.

[scott-white-td]

@gdavison @varunrsystems - we saw this error in US-east-1 and eu-central-1 on April 28.

I would guess AWS is rolling out the same change to all regions, one by one.

I'm using provider v3.37.0

[scott-white-td]

When will v3.38.0 be released? This error is blocking some production updates we need to deploy via terraform.

[dayglojesus]

This is a me too. Stack is in us-west-2 and it was fine last week, but this week it's been balking anytime we plan it ...

Error: error reading Storage Gateway Bandwidth rate limit: InvalidGatewayRequestException: This operation is not valid for the specified gateway.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "dd514340-7555-4039-b046-e597a4bbaa41"
  },
  Error_: {
    ErrorCode: "UnsupportedOperationForGatewayType"
  },
  Message_: "This operation is not valid for the specified gateway."
}

[nerryc]

We have also tried many different providers, ended up having to remove the storage gateway from state and add the arn values called statically just to continue our runs.. Logged a call with HashiCorp about this as well.

[rnebular]

We are experiencing this as well in the us-west-2 region. Plan/Apply worked two weeks ago on existing infrastructure. Now existing plan/apply failing as well as new undeployed gateways also showing this error during plan/apply. I would copy/paste our error but it's identical to others that have already posted. This is 100% blocking, and we've tried to hard-code to an older provider but have found no relief.

Waiting for a fix or workaround...

[varunrsystems]

If anyone having AWS enterprise support then they can also raise this issue as production block on AWS, I already raised but not get response from last 2 days.

[nerryc]

We raised it and was basically told to fix the api call being made... so in this scenario terraform needs to adjust.

Pretty poor show from AWS not advertising these changes.

This is also not currently happening in all regions, adding to the confusion and frustration.

[ghost]

This has been released in version 3.38.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[varunrsystems]

Update from AWS and I guess they are right about how to handle the exception.

Thank you for bringing this issue to our attention. Please note that the DescribeBandwidthRateLimit API is not supported on the Amazon S3 File Gateway. Therefore, calling the API with this gateway type is expected to throw an InvalidGatewayRequestException error.

We are aware that some Terraform configuration scripts to create Storage Gateway resources appear to call the DescribeBandwidthRateLimit API for all gateway types. It appears that the exception handling that is intended to handle errors from DescribeBandwidthRateLimit with unsupported gateway types, utilizes both the error and message fields to parse the exception. Since we have updated the associated error message for this API, the script appears to fail as a result.

In response to your message, we have taken steps to update the error message in order to address the issue as quickly as possible. However, please note that we do not recommend utilizing the message field from API errors for exception handling but instead recommend only using the error field in order to avoid similar issues in the future.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.