Add table gcp_dataproc_cluster closes #376 #379

ParthaI · 2022-08-10T08:17:56Z

Integration test logs

Logs

No env file present for the current environment:  staging 
 Falling back to .env config
No env file present for the current environment:  staging
customEnv TURBOT_TEST_EXPECTED_TIMEOUT undefined

SETUP: tests/gcp_dataproc_cluster []

PRETEST: tests/gcp_dataproc_cluster

TEST: tests/gcp_dataproc_cluster
Running terraform
data.google_client_config.current: Reading...
data.google_client_config.current: Read complete after 0s [id=projects/parker-aaa/regions/us-east1/zones/]
data.null_data_source.resource: Reading...
data.null_data_source.resource: Read complete after 0s [id=static]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_dataproc_cluster.named_test_resource will be created
  + resource "google_dataproc_cluster" "named_test_resource" {
      + graceful_decommission_timeout = "0s"
      + id                            = (known after apply)
      + labels                        = (known after apply)
      + name                          = "turbottest91470"
      + project                       = (known after apply)
      + region                        = "us-east1"

      + cluster_config {
          + bucket      = (known after apply)
          + temp_bucket = (known after apply)

          + endpoint_config {
              + enable_http_port_access = (known after apply)
              + http_ports              = (known after apply)
            }

          + gce_cluster_config {
              + internal_ip_only       = false
              + network                = (known after apply)
              + service_account_scopes = (known after apply)
              + subnetwork             = "projects/parker-aaa/regions/us-east1/subnetworks/test21"
              + tags                   = [
                  + "bar",
                  + "foo",
                ]
              + zone                   = "us-east1-b"

              + shielded_instance_config {
                  + enable_integrity_monitoring = (known after apply)
                  + enable_secure_boot          = (known after apply)
                  + enable_vtpm                 = (known after apply)
                }
            }

          + master_config {
              + image_uri        = (known after apply)
              + instance_names   = (known after apply)
              + machine_type     = (known after apply)
              + min_cpu_platform = (known after apply)
              + num_instances    = (known after apply)

              + accelerators {
                  + accelerator_count = (known after apply)
                  + accelerator_type  = (known after apply)
                }

              + disk_config {
                  + boot_disk_size_gb = (known after apply)
                  + boot_disk_type    = (known after apply)
                  + num_local_ssds    = (known after apply)
                }
            }

          + preemptible_worker_config {
              + instance_names = (known after apply)
              + num_instances  = (known after apply)
              + preemptibility = (known after apply)

              + disk_config {
                  + boot_disk_size_gb = (known after apply)
                  + boot_disk_type    = (known after apply)
                  + num_local_ssds    = (known after apply)
                }
            }

          + software_config {
              + image_version       = (known after apply)
              + optional_components = (known after apply)
              + override_properties = (known after apply)
              + properties          = (known after apply)
            }

          + worker_config {
              + image_uri        = (known after apply)
              + instance_names   = (known after apply)
              + machine_type     = (known after apply)
              + min_cpu_platform = (known after apply)
              + num_instances    = (known after apply)

              + accelerators {
                  + accelerator_count = (known after apply)
                  + accelerator_type  = (known after apply)
                }

              + disk_config {
                  + boot_disk_size_gb = (known after apply)
                  + boot_disk_type    = (known after apply)
                  + num_local_ssds    = (known after apply)
                }
            }
        }

      + virtual_cluster_config {
          + staging_bucket = (known after apply)

          + auxiliary_services_config {
              + metastore_config {
                  + dataproc_metastore_service = (known after apply)
                }

              + spark_history_server_config {
                  + dataproc_cluster = (known after apply)
                }
            }

          + kubernetes_cluster_config {
              + kubernetes_namespace = (known after apply)

              + gke_cluster_config {
                  + gke_cluster_target = (known after apply)

                  + node_pool_target {
                      + node_pool = (known after apply)
                      + roles     = (known after apply)

                      + node_pool_config {
                          + locations = (known after apply)

                          + autoscaling {
                              + max_node_count = (known after apply)
                              + min_node_count = (known after apply)
                            }

                          + config {
                              + local_ssd_count  = (known after apply)
                              + machine_type     = (known after apply)
                              + min_cpu_platform = (known after apply)
                              + preemptible      = (known after apply)
                              + spot             = (known after apply)
                            }
                        }
                    }
                }

              + kubernetes_software_config {
                  + component_version = (known after apply)
                  + properties        = (known after apply)
                }
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + project_id    = "parker-aaa"
  + resource_aka  = (known after apply)
  + resource_id   = (known after apply)
  + resource_name = "turbottest91470"
google_dataproc_cluster.named_test_resource: Creating...
google_dataproc_cluster.named_test_resource: Still creating... [10s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [20s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [30s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [40s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [50s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [1m0s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [1m10s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [1m20s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [1m30s elapsed]
google_dataproc_cluster.named_test_resource: Still creating... [1m40s elapsed]
google_dataproc_cluster.named_test_resource: Creation complete after 1m49s [id=projects/parker-aaa/regions/us-east1/clusters/turbottest91470]

Warning: Deprecated Resource

  with data.null_data_source.resource,
  on variables.tf line 27, in data "null_data_source" "resource":
  27: data "null_data_source" "resource" {

The null_data_source was historically used to construct intermediate values
to re-use elsewhere in configuration, the same can now be achieved using
locals

(and one more similar warning elsewhere)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

project_id = "parker-aaa"
resource_aka = "gcp://dataproc.googleapis.com/projects/parker-aaa/regions/us-east1/clusters/turbottest91470"
resource_id = "projects/parker-aaa/regions/us-east1/clusters/turbottest91470"
resource_name = "turbottest91470"

Running SQL query: test-get-query.sql
[
  {
    "cluster_name": "turbottest91470",
    "project": "parker-aaa",
    "title": "turbottest91470"
  }
]
✔ PASSED

Running SQL query: test-list-query.sql
[
  {
    "akas": [
      "gcp://dataproc.googleapis.com/projects/parker-aaa/regions/us-east1/clusters/turbottest91470"
    ],
    "cluster_name": "turbottest91470",
    "project": "parker-aaa",
    "title": "turbottest91470"
  }
]
✔ PASSED

Running SQL query: test-notfound-query.sql
null
✔ PASSED

Running SQL query: test-turbot-query.sql
[
  {
    "akas": [
      "gcp://dataproc.googleapis.com/projects/parker-aaa/regions/us-east1/clusters/turbottest91470"
    ],
    "title": "turbottest91470"
  }
]
✔ PASSED

POSTTEST: tests/gcp_dataproc_cluster

TEARDOWN: tests/gcp_dataproc_cluster

SUMMARY:

1/1 passed.

Example query results

Results

> select * from gcp_dataproc_cluster;
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------
| cluster_name        | cluster_uuid                         | state | config                                                                                                   
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------
| cluster-2261-delete | 9c68c754-3062-49d8-9962-093c5ef7d477 | ERROR | {"configBucket":"dataproc-81d7c6b8-6d8c-4b04-af65-1bf0751bd6fd-us-east1","encryptionConfig":{"gcePdKmsKey
|                     |                                      |       | -2-0-deb10-20220722-050200-rc01","instanceNames":["cluster-2261-delete-m"],"machineTypeUri":"https://www.
|                     |                                      |       | de.handler.count":"20","hdfs:dfs.namenode.http-address":"0.0.0.0:9870","hdfs:dfs.namenode.https-address":
|                     |                                      |       | ","mapred:mapreduce.task.io.sort.mb":"256","mapred:yarn.app.mapreduce.am.command-opts":"-Xmx2524m","mapre
|                     |                                      |       | nager-graceful-decommission-timeout-secs":"86400","yarn:yarn.scheduler.maximum-allocation-mb":"12624","ya
| cluster-5824        | c1ce399e-54d9-4aab-bf95-e1f2362ad426 | ERROR | {"configBucket":"dataproc-81d7c6b8-6d8c-4b04-af65-1bf0751bd6fd-us-east1","encryptionConfig":{},"endpointC
|                     |                                      |       | s://www.googleapis.com/compute/v1/projects/parker-aaa/zones/us-east1-b/machineTypes/e2-standard-2","minCp
|                     |                                      |       | ress":"0.0.0.0:9871","hdfs:dfs.namenode.lifeline.rpc-address":"cluster-5824-m:8050","hdfs:dfs.namenode.se
|                     |                                      |       | p.mapreduce.am.resource.cpu-vcores":"1","mapred:yarn.app.mapreduce.am.resource.mb":"3277","spark-env:SPAR
|                     |                                      |       | eduler.minimum-allocation-mb":"1"}},"tempBucket":"dataproc-temp-us-east1-979620418102-hkzrurqz","workerCo
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------
> select * from gcp_dataproc_cluster where cluster_name = 'cluster-5824'
+--------------+--------------------------------------+-------+-----------------------------------------------------------------------------------------------------------------
| cluster_name | cluster_uuid                         | state | config                                                                                                          
+--------------+--------------------------------------+-------+-----------------------------------------------------------------------------------------------------------------
| cluster-5824 | c1ce399e-54d9-4aab-bf95-e1f2362ad426 | ERROR | {"configBucket":"dataproc-81d7c6b8-6d8c-4b04-af65-1bf0751bd6fd-us-east1","encryptionConfig":{},"endpointConfig":
|              |                                      |       | s://www.googleapis.com/compute/v1/projects/parker-aaa/zones/us-east1-b/machineTypes/e2-standard-2","minCpuPlatfo
|              |                                      |       | ress":"0.0.0.0:9871","hdfs:dfs.namenode.lifeline.rpc-address":"cluster-5824-m:8050","hdfs:dfs.namenode.secondary
|              |                                      |       | p.mapreduce.am.resource.cpu-vcores":"1","mapred:yarn.app.mapreduce.am.resource.mb":"3277","spark-env:SPARK_DAEMO
|              |                                      |       | eduler.minimum-allocation-mb":"1"}},"tempBucket":"dataproc-temp-us-east1-979620418102-hkzrurqz","workerConfig":{
+--------------+--------------------------------------+-------+-----------------------------------------------------------------------------------------------------------------
> select * from gcp_dataproc_cluster where state = 'ERROR'
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------
| cluster_name        | cluster_uuid                         | state | config                                                                                                   
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------
| cluster-2261-delete | 9c68c754-3062-49d8-9962-093c5ef7d477 | ERROR | {"configBucket":"dataproc-81d7c6b8-6d8c-4b04-af65-1bf0751bd6fd-us-east1","encryptionConfig":{"gcePdKmsKey
|                     |                                      |       | -2-0-deb10-20220722-050200-rc01","instanceNames":["cluster-2261-delete-m"],"machineTypeUri":"https://www.
|                     |                                      |       | de.handler.count":"20","hdfs:dfs.namenode.http-address":"0.0.0.0:9870","hdfs:dfs.namenode.https-address":
|                     |                                      |       | ","mapred:mapreduce.task.io.sort.mb":"256","mapred:yarn.app.mapreduce.am.command-opts":"-Xmx2524m","mapre
|                     |                                      |       | nager-graceful-decommission-timeout-secs":"86400","yarn:yarn.scheduler.maximum-allocation-mb":"12624","ya
| cluster-5824        | c1ce399e-54d9-4aab-bf95-e1f2362ad426 | ERROR | {"configBucket":"dataproc-81d7c6b8-6d8c-4b04-af65-1bf0751bd6fd-us-east1","encryptionConfig":{},"endpointC
|                     |                                      |       | s://www.googleapis.com/compute/v1/projects/parker-aaa/zones/us-east1-b/machineTypes/e2-standard-2","minCp
|                     |                                      |       | ress":"0.0.0.0:9871","hdfs:dfs.namenode.lifeline.rpc-address":"cluster-5824-m:8050","hdfs:dfs.namenode.se
|                     |                                      |       | p.mapreduce.am.resource.cpu-vcores":"1","mapred:yarn.app.mapreduce.am.resource.mb":"3277","spark-env:SPAR
|                     |                                      |       | eduler.minimum-allocation-mb":"1"}},"tempBucket":"dataproc-temp-us-east1-979620418102-hkzrurqz","workerCo
+---------------------+--------------------------------------+-------+----------------------------------------------------------------------------------------------------------

bigdatasourav · 2022-08-12T13:56:16Z

docs/tables/gcp_dataproc_cluster.md

@@ -0,0 +1,44 @@
+# Table: gcp_compute_zone


Suggested change

# Table: gcp_compute_zone

# Table: gcp_dataproc_cluster

bigdatasourav · 2022-08-12T13:57:58Z

gcp-test/tests/gcp_dataproc_cluster/variables.tf

+  }
+}
+
+# resource "google_compute_network" "named_test_resource" {


Can we remove this commented part?

bigdatasourav · 2022-08-12T14:03:15Z

gcp/table_gcp_dataproc_cluster.go

+	}
+}
+
+type ClusterInfo struct {


Please remove this empty structure

bigdatasourav

LGTM

misraved · 2022-08-16T12:09:40Z

gcp-test/tests/gcp_dataproc_cluster/variables.tf

+
+      # One of the below to hook into a custom network / subnetwork
+      # network    = google_compute_network.named_test_resource.name
+      subnetwork = "projects/parker-aaa/regions/us-east1/subnetworks/test21"


Why has the subnetwork been hard-coded?

docs/tables/gcp_dataproc_cluster.md

…to issue-376

Co-authored-by: Ved misra <47312748+misraved@users.noreply.github.com>

ParthaI added 2 commits August 10, 2022 11:04

Add table gcp_dataproc_cluster closes #376

708bba2

added the intg test

220c58a

ParthaI requested a review from bigdatasourav August 10, 2022 08:17

ParthaI self-assigned this Aug 10, 2022

ParthaI linked an issue Aug 10, 2022 that may be closed by this pull request

Add table gcp_dataproc_cluster #376

Closed

Updated the log statement

17d0cad

bigdatasourav reviewed Aug 12, 2022

View reviewed changes

gcp/table_gcp_dataproc_cluster.go Outdated

}

}

type ClusterInfo struct {

Copy link

Contributor

bigdatasourav Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this empty structure

ParthaI and others added 5 commits August 16, 2022 12:00

Updated the table doc

bf200eb

Removed the commented code

9649cf6

Removed the empty pre define struct which was not being used

622f1f9

Formated the code

29e5826

Update table_gcp_dataproc_cluster.go

9b05fa2

bigdatasourav approved these changes Aug 16, 2022

View reviewed changes

misraved reviewed Aug 16, 2022

View reviewed changes

docs/tables/gcp_dataproc_cluster.md Outdated Show resolved Hide resolved

ParthaI and others added 4 commits August 16, 2022 18:47

Updated the resource configuration in tf

20e3638

Merge branch 'issue-376' of github.com:turbot/steampipe-plugin-gcp in…

6890cc3

…to issue-376

Update docs/tables/gcp_dataproc_cluster.md

fa56232

Co-authored-by: Ved misra <47312748+misraved@users.noreply.github.com>

Update variables.tf

cabd82e

misraved approved these changes Aug 16, 2022

View reviewed changes

misraved merged commit 463029a into main Aug 16, 2022

misraved deleted the issue-376 branch August 16, 2022 13:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add table gcp_dataproc_cluster closes #376 #379

Add table gcp_dataproc_cluster closes #376 #379

ParthaI commented Aug 10, 2022

bigdatasourav Aug 12, 2022

bigdatasourav Aug 12, 2022

bigdatasourav Aug 12, 2022

bigdatasourav left a comment

misraved Aug 16, 2022

Add table gcp_dataproc_cluster closes #376 #379

Add table gcp_dataproc_cluster closes #376 #379

Conversation

ParthaI commented Aug 10, 2022

Integration test logs

Example query results

bigdatasourav Aug 12, 2022

Choose a reason for hiding this comment

bigdatasourav Aug 12, 2022

Choose a reason for hiding this comment

bigdatasourav Aug 12, 2022

Choose a reason for hiding this comment

bigdatasourav left a comment

Choose a reason for hiding this comment

misraved Aug 16, 2022

Choose a reason for hiding this comment