[BUG]: github_rest_api parses body inconsistently #1953

raihle · 2023-10-12T07:47:50Z

Expected Behavior

I expect to get the whole response in the "body" of the terraform data source.

Actual Behavior

I sometimes get a partial result, sometimes null. I have tried calling the API manually and using mitmproxy to intercept the data that is returned from GitHub when running terraform, and it is always complete. With TF_LOG=debug it seems that the provider is also getting a full response (see below).

Terraform Version

Terraform v1.5.5
on darwin_arm64

provider registry.terraform.io/hashicorp/random v3.5.1
provider registry.terraform.io/integrations/github v5.39.0
provider registry.terraform.io/mastercard/restapi v1.18.2

Affected Resource(s)

github_rest_api

Terraform Configuration Files

terraform {
  required_version = " >= 1.5"
  required_providers {
    github = {
      source  = "integrations/github"
      version = ">= 5.39.0"
    }
  }
}

variable "github_token" {
  type        = string
  description = "An API token for GitHub"
  sensitive   = true
}

variable "github_organization" {
  type        = string
  description = "The login name of an organization to look up"
}

data "github_rest_api" "organization" {
  endpoint = "orgs/${var.github_organization}"
}

output "organization" {
  value = data.github_rest_api.organization.body
}

Steps to Reproduce

terraform apply

Debug Output

2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: 2023/10/12 10:36:18 [DEBUG] GitHub API Response Details:
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: ---[ RESPONSE ]--------------------------------------
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: HTTP/2.0 200 OK
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Access-Control-Allow-Origin: *
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Cache-Control: private, max-age=60, s-maxage=60
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Content-Security-Policy: default-src 'none'
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Content-Type: application/json; charset=utf-8
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Date: Thu, 12 Oct 2023 07:36:18 GMT
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Etag: W/"4a467e665db2f1c6d98fd445e272b2f0baa3a03a52d535bdf70c437d193c1cee"
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Github-Authentication-Token-Expiration: 2023-11-04 13:18:34 UTC
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Last-Modified: Fri, 02 Dec 2022 22:54:17 GMT
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Server: GitHub.com
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Vary: Accept, Authorization, Cookie, X-GitHub-OTP
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Vary: Accept-Encoding, Accept, X-Requested-With
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Accepted-Oauth-Scopes: admin:org, read:org, repo, user, write:org
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Content-Type-Options: nosniff
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Frame-Options: deny
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Api-Version-Selected: 2022-11-28
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Media-Type: github.v3; format=json, github.stone-crop-preview; format=json
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Request-Id: 0892:843E:6EDAC69:70024BA:6527A1F2
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Oauth-Scopes: repo, user:email
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Limit: 5000
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Remaining: 4995
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Reset: 1697099744
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Resource: core
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Used: 5
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Xss-Protection: 0
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: 
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: {
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "login": "org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "id": 00000000,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "node_id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "url": "https://api.github.com/orgs/org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "repos_url": "https://api.github.com/orgs/org-login/repos",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "events_url": "https://api.github.com/orgs/org-login/events",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "hooks_url": "https://api.github.com/orgs/org-login/hooks",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "issues_url": "https://api.github.com/orgs/org-login/issues",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_url": "https://api.github.com/orgs/org-login/members{/member}",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_members_url": "https://api.github.com/orgs/org-login/public_members{/member}",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "avatar_url": "https://avatars.githubusercontent.com/u/00000000?v=4",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "description": "Organization Description",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "name": "Organization Name",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "company": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "blog": "url",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "location": "country",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "email": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "twitter_username": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "is_verified": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "has_organization_projects": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "has_repository_projects": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_gists": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "followers": 409,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "following": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "html_url": "https://github.com/org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "created_at": "2021-02-08T10:08:12Z",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "updated_at": "2022-12-02T22:54:17Z",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "archived_at": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "type": "Organization",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "total_private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "owned_private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "private_gists": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "disk_usage": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "collaborators": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "billing_email": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "default_repository_permission": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "two_factor_requirement_enabled": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_allowed_repository_creation_type": "private",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_public_repositories": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_private_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_internal_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_pages": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_fork_private_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "web_commit_signoff_required": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_public_pages": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_private_pages": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "plan": {
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "name": "enterprise",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "space": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "filled_seats": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "seats": 0
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  }
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: }
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: -----------------------------------------------------
module.first_scope.data.github_rest_api.organization: Read complete after 0s [id=0892:843E:6EDAC69:70024BA:6527A1F2]

Panic Output

Not a panic per se, but the output of the module above is inconsistent (but always incomplete).

Some examples:

organization = tomap({
  "description" = "Organization Description"
  "private_gists" = ""
  "type" = "Organization"
})

organization = null

organization = tomap({
  "node_id" = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  "repos_url" = "https://api.github.com/orgs/login/repos"
})

organization = tomap({
  "disk_usage" = ""
  "login" = "login"
  "node_id" = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  "private_gists" = ""
})

No changes were made between the runs above - I'm simply rerunning terraform apply

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

SharpEdgeMarshall · 2024-02-15T15:08:54Z

This is critical for us

riezebosch · 2024-02-16T08:46:12Z

For me as well, probably related: #1776. Tried out this fix #2110 using 6.0.0-rc2 but still seeing null in the state file.

raihle added Status: Triage This is being looked at and prioritized Type: Bug Something isn't working as documented labels Oct 12, 2023

kfcampbell added Status: Up for grabs Issues that are ready to be worked on by anyone and removed Status: Triage This is being looked at and prioritized labels Oct 16, 2023

This was referenced Feb 16, 2024

fix: github_rest_api data source always returns header and body as null #2110

Merged

fix(data_source_github_rest_api): just read body and convert bytes into string #2152

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: github_rest_api parses body inconsistently #1953

[BUG]: github_rest_api parses body inconsistently #1953

raihle commented Oct 12, 2023 •

edited

SharpEdgeMarshall commented Feb 15, 2024

riezebosch commented Feb 16, 2024

[BUG]: github_rest_api parses body inconsistently #1953

[BUG]: github_rest_api parses body inconsistently #1953

Comments

raihle commented Oct 12, 2023 • edited

Expected Behavior

Actual Behavior

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Code of Conduct

SharpEdgeMarshall commented Feb 15, 2024

riezebosch commented Feb 16, 2024

raihle commented Oct 12, 2023 •

edited