External metastores support for most HDInsight resources #6969

kosinsky · 2020-05-17T16:22:30Z

Depends On #6809

PR #6145 added support for external metastores for azurerm_hdinsight_hadoop_cluster. Using the same approach to expand support for:

azurerm_hdinsight_hbase_cluster
azurerm_hdinsight_interactive_query_cluster
azurerm_hdinsight_kafka_cluster
azurerm_hdinsight_spark_cluster
azurerm_hdinsight_storm_cluster

Sample configuration:

metastores {
    hive {
        server   = azurerm_sql_server.test.fully_qualified_domain_name
        database_name = azurerm_sql_database.test.name
        username      = azurerm_sql_server.test.administrator_login
        password      = azurerm_sql_server.test.administrator_login_password
    }
    oozie {
        server   = azurerm_sql_server.test.fully_qualified_domain_name
        database_name = azurerm_sql_database.test.name
        username      = azurerm_sql_server.test.administrator_login
        password      = azurerm_sql_server.test.administrator_login_password
    }
    ambari {
        server   = azurerm_sql_server.test.fully_qualified_domain_name
        database_name = azurerm_sql_database.test.name
        username      = azurerm_sql_server.test.administrator_login
        password      = azurerm_sql_server.test.administrator_login_password
   }
}

New tests:

make testacc TEST='./azurerm/internal/services/hdinsight/tests/' TESTARGS='-run=Metastore -parallel=2' TESTTIMEOUT='300m'                                           
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
TF_ACC=1 go test ./azurerm/internal/services/hdinsight/tests/ -v -run=Metastore -parallel=2 -timeout 300m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightHadoopCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_updateMetastore
=== RUN   TestAccAzureRMHDInsightHBaseCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightHBaseCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_updateMetastore
=== RUN   TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
=== RUN   TestAccAzureRMHDInsightKafkaCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightKafkaCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_updateMetastore
=== RUN   TestAccAzureRMHDInsightSparkCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightSparkCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightSparkCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightSparkCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightSparkCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightSparkCluster_updateMetastore
=== RUN   TestAccAzureRMHDInsightStormCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightStormCluster_allMetastores
=== RUN   TestAccAzureRMHDInsightStormCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightStormCluster_hiveMetastore
=== RUN   TestAccAzureRMHDInsightStormCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightStormCluster_updateMetastore
=== CONT  TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== CONT  TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightKafkaCluster_hiveMetastore (1121.89s)
=== CONT  TestAccAzureRMHDInsightStormCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightStormCluster_allMetastores (898.22s)
=== CONT  TestAccAzureRMHDInsightStormCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_allMetastores (2111.41s)
=== CONT  TestAccAzureRMHDInsightStormCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightStormCluster_hiveMetastore (889.58s)
=== CONT  TestAccAzureRMHDInsightHBaseCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightStormCluster_updateMetastore (1651.68s)
=== CONT  TestAccAzureRMHDInsightKafkaCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightKafkaCluster_allMetastores (1065.09s)
=== CONT  TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_updateMetastore (2024.42s)
=== CONT  TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore (1014.18s)
=== CONT  TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore (2084.51s)
=== CONT  TestAccAzureRMHDInsightSparkCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores (1081.28s)
=== CONT  TestAccAzureRMHDInsightHBaseCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightSparkCluster_hiveMetastore (1182.48s)
=== CONT  TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_allMetastores (1008.87s)
=== CONT  TestAccAzureRMHDInsightSparkCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_hiveMetastore (1056.78s)
=== CONT  TestAccAzureRMHDInsightHadoopCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightSparkCluster_updateMetastore (1892.70s)
=== CONT  TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_updateMetastore (1772.36s)
=== CONT  TestAccAzureRMHDInsightKafkaCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_hiveMetastore (1075.86s)
=== CONT  TestAccAzureRMHDInsightSparkCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightSparkCluster_allMetastores (1066.44s)
--- PASS: TestAccAzureRMHDInsightKafkaCluster_updateMetastore (2011.75s)
PASS
ok      github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/hdinsight/tests     12845.822s

I skipped MLService/RServer because they a pretty unstable (every other time even basic test fails with Code="BadRequest" Message="HDI Version'3.6' is not supported for clusterType 'MLServices' and componentVersion 'default'.") and scheduled to retire by the end of 2020:

The ML Services cluster type on HDInsight is supported only on HDInsight 3.6. HDInsight 3.6 is scheduled to retire on December 31, 2020.

Fixes #5230 for most HDInsight resources

kosinsky · 2020-05-21T14:55:11Z

@jackofallops, could you take a look? This PR is applying #6145 approach for other HDInsight resources.

ross-p-smith · 2020-06-03T11:54:10Z

Hi @jackofallops - I require this change for a project I am working on. Do you know a rough ETA so that I can plan my sprints? Many thanks for any help you can offer

jackofallops

Hi @kosinsky
Thanks for this PR extending your previous work, and apologies for the delay in getting to this review.

It's all looking good, but there's one specific change required in each cluster case, noted below. (I missed the use of GetOkExists in the last PR, sorry).

The Plugin SDK upgrade PR is in progress, and we're hoping to get it merged asap, so this will need to be rebased after that goes in to avoid conflicts. (I've added the dependency note to the PR desc). I'll circle back to re-review and run the acceptance tests at that point.

Thanks again!

Ste

azurerm/internal/services/hdinsight/hdinsight_hbase_cluster_resource.go

azurerm/internal/services/hdinsight/hdinsight_interactive_query_cluster_resource.go

azurerm/internal/services/hdinsight/hdinsight_kafka_cluster_resource.go

azurerm/internal/services/hdinsight/hdinsight_spark_cluster_resource.go

azurerm/internal/services/hdinsight/hdinsight_storm_cluster_resource.go

kosinsky · 2020-06-12T23:24:37Z

@jackofallops Done.

make testacc TEST='./azurerm/internal/services/hdinsight/tests/' TESTARGS='-run=Metastore -parallel=2' TESTTIMEOUT='300m'
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
TF_ACC=1 go test ./azurerm/internal/services/hdinsight/tests/ -v -run=Metastore -parallel=2 -timeout 300m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== RUN TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightHadoopCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightHadoopCluster_updateMetastore
=== RUN TestAccAzureRMHDInsightHBaseCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_allMetastores
=== RUN TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightHBaseCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightHBaseCluster_updateMetastore
=== RUN TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
=== RUN TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
=== RUN TestAccAzureRMHDInsightKafkaCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_allMetastores
=== RUN TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightKafkaCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightKafkaCluster_updateMetastore
=== RUN TestAccAzureRMHDInsightSparkCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightSparkCluster_allMetastores
=== RUN TestAccAzureRMHDInsightSparkCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightSparkCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightSparkCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightSparkCluster_updateMetastore
=== RUN TestAccAzureRMHDInsightStormCluster_allMetastores
=== PAUSE TestAccAzureRMHDInsightStormCluster_allMetastores
=== RUN TestAccAzureRMHDInsightStormCluster_hiveMetastore
=== PAUSE TestAccAzureRMHDInsightStormCluster_hiveMetastore
=== RUN TestAccAzureRMHDInsightStormCluster_updateMetastore
=== PAUSE TestAccAzureRMHDInsightStormCluster_updateMetastore
=== CONT TestAccAzureRMHDInsightHadoopCluster_allMetastores
=== CONT TestAccAzureRMHDInsightKafkaCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_allMetastores (1091.92s)
=== CONT TestAccAzureRMHDInsightHBaseCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightKafkaCluster_hiveMetastore (1130.79s)
=== CONT TestAccAzureRMHDInsightKafkaCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightKafkaCluster_allMetastores (1131.90s)
=== CONT TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_updateMetastore (2515.60s)
=== CONT TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_updateMetastore (1969.68s)
=== CONT TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_hiveMetastore (1117.55s)
=== CONT TestAccAzureRMHDInsightStormCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightInteractiveQueryCluster_allMetastores (1278.22s)
=== CONT TestAccAzureRMHDInsightStormCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightStormCluster_allMetastores (1043.68s)
=== CONT TestAccAzureRMHDInsightStormCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightStormCluster_hiveMetastore (943.80s)
=== CONT TestAccAzureRMHDInsightHBaseCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightStormCluster_updateMetastore (1786.09s)
=== CONT TestAccAzureRMHDInsightHBaseCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_allMetastores (1132.15s)
=== CONT TestAccAzureRMHDInsightHadoopCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHBaseCluster_hiveMetastore (1023.06s)
=== CONT TestAccAzureRMHDInsightSparkCluster_allMetastores
--- PASS: TestAccAzureRMHDInsightSparkCluster_allMetastores (956.13s)
=== CONT TestAccAzureRMHDInsightKafkaCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_updateMetastore (1748.23s)
=== CONT TestAccAzureRMHDInsightSparkCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightSparkCluster_hiveMetastore (1012.98s)
=== CONT TestAccAzureRMHDInsightSparkCluster_updateMetastore
--- PASS: TestAccAzureRMHDInsightKafkaCluster_updateMetastore (2028.44s)
=== CONT TestAccAzureRMHDInsightHadoopCluster_hiveMetastore
--- PASS: TestAccAzureRMHDInsightHadoopCluster_hiveMetastore (1080.97s)
--- PASS: TestAccAzureRMHDInsightSparkCluster_updateMetastore (1965.47s)
PASS
ok github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/hdinsight/tests 12573.878s

tombuildsstuff · 2020-07-06T11:00:24Z

👋

Just to give an update here, whilst this PR was dependent on #6809 - we ended up disabling Binary Testing in that PR - work to enable Binary testing will happen in a subsequent PR (and is being investigated) - however unfortunately this PR is still blocked for the moment.

Thanks!

kosinsky · 2020-07-06T14:15:01Z

Thanks for letting me know. However, I'm not sure how it depends on on #6809. This PR is extenting of #6145 to all HDI resources and #6145 was merged in 2.9 time frame.

jackofallops · 2020-07-06T14:20:25Z

@kosinsky - Apologies for the confusion, we had a crossed wire somewhere. This PR was blocked by 6809, but for the GetOkExists deprecation, not the binary testing issue we're working on. I've got this PR open now and working through it to resolve the merge conflict and get the testing kicked off on our CI server in preparation for merge.

Thanks again!

Update: full regression test is running now.

…GetExistsOk` is going away

This reverts commit 1bafc3d.

This reverts commit a4166b0.

This reverts commit 30536e7.

Co-authored-by: Steve <11830746+jackofallops@users.noreply.github.com>

jackofallops · 2020-07-08T10:11:36Z

Hi @kosinsky - I've had to do a bit of git wrangling to account for a mistake we had to correct on master creating a bit of a diverge in history. I think I've got it straight now, but if you can check everything looks correct to you I'll merge later (pending the tests on our CI, ofc)

Thanks!

jackofallops · 2020-07-08T11:30:46Z

Test passing (failures are transient)

kosinsky · 2020-07-08T11:37:39Z

@jackofallops, Looks good for me. I ran some acceptance tests (new and *_basic) they are green too.

jackofallops · 2020-07-08T12:45:45Z

@jackofallops, Looks good for me. I ran some acceptance tests (new and *_basic) they are green too.

Brilliant, thanks! I'll merge this then follow up on your other PR for monitor next.

ghost · 2020-07-10T01:23:20Z

This has been released in version 2.18.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.18.0"
}
# ... other configuration ...

ghost · 2020-08-07T14:51:01Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

ghost added size/XXL documentation labels May 17, 2020

jackofallops added enhancement service/hdinsight labels Jun 12, 2020

jackofallops requested changes Jun 12, 2020

View reviewed changes

jackofallops added this to the v2.16.0 milestone Jun 18, 2020

jackofallops modified the milestones: v2.16.0, v2.17.0 Jun 25, 2020

tombuildsstuff modified the milestones: v2.17.0, v2.18.0 Jul 2, 2020

jackofallops force-pushed the external_mtastore branch from 5c1acca to 87e5c45 Compare July 6, 2020 14:40

tombuildsstuff force-pushed the master branch from 5ae47f9 to 68a2f63 Compare July 6, 2020 16:30

jackofallops force-pushed the external_mtastore branch from 3b78303 to 87e5c45 Compare July 8, 2020 07:24

tombuildsstuff and others added 11 commits July 8, 2020 09:03

r/postgresql_server: requiring the use of ssl_enforcement since `d.…

f7bfeaf

…GetExistsOk` is going away

make lint

6cd20f7

External metastores for HBase

8898d9b

External metastores for Interactive Hive

c68b354

External metastores for Kafka

3d7612b

External metastores for MLServices

456aeba

External metastores for RServer

b90be9a

EXternal metastores for Spark

f3ddcb2

Error message fixes

711f683

External metastore for Storm

b0357be

Fix for msservices / rserver

9a7db8a

kosinsky and others added 8 commits July 8, 2020 09:04

Revert "Fix for msservices / rserver"

a36c73b

This reverts commit 1bafc3d.

Revert "External metastores for RServer"

b8943f7

This reverts commit a4166b0.

Revert "External metastores for MLServices"

3545c58

This reverts commit 30536e7.

Documentation for external metastores

bd5c6d3

Remove GetOkExists

5e2c5a9

Co-authored-by: Steve <11830746+jackofallops@users.noreply.github.com>

Fix typo in hbase

41abfd7

Get rid of GetOkExists in all hdinsight resources

69c1fba

merge conflict resolution and minor change for metastoreraw

a01ee6f

jackofallops force-pushed the external_mtastore branch from 87e5c45 to a01ee6f Compare July 8, 2020 08:09

deps

bd8bef0

ghost added the dependencies label Jul 8, 2020

jackofallops approved these changes Jul 8, 2020

View reviewed changes

jackofallops merged commit a1519f1 into hashicorp:master Jul 8, 2020

jackofallops added a commit that referenced this pull request Jul 8, 2020

Update CHANGELOG.md for #6969

3d7e105

hashicorp locked and limited conversation to collaborators Aug 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External metastores support for most HDInsight resources #6969

External metastores support for most HDInsight resources #6969

kosinsky commented May 17, 2020 •

edited by jackofallops

kosinsky commented May 21, 2020

ross-p-smith commented Jun 3, 2020

jackofallops left a comment

kosinsky commented Jun 12, 2020

tombuildsstuff commented Jul 6, 2020

kosinsky commented Jul 6, 2020

jackofallops commented Jul 6, 2020 •

edited

jackofallops commented Jul 8, 2020

jackofallops commented Jul 8, 2020

kosinsky commented Jul 8, 2020 •

edited

jackofallops commented Jul 8, 2020

ghost commented Jul 10, 2020

ghost commented Aug 7, 2020

External metastores support for most HDInsight resources #6969

External metastores support for most HDInsight resources #6969

Conversation

kosinsky commented May 17, 2020 • edited by jackofallops

kosinsky commented May 21, 2020

ross-p-smith commented Jun 3, 2020

jackofallops left a comment

Choose a reason for hiding this comment

kosinsky commented Jun 12, 2020

tombuildsstuff commented Jul 6, 2020

kosinsky commented Jul 6, 2020

jackofallops commented Jul 6, 2020 • edited

jackofallops commented Jul 8, 2020

jackofallops commented Jul 8, 2020

kosinsky commented Jul 8, 2020 • edited

jackofallops commented Jul 8, 2020

ghost commented Jul 10, 2020

ghost commented Aug 7, 2020

kosinsky commented May 17, 2020 •

edited by jackofallops

jackofallops commented Jul 6, 2020 •

edited

kosinsky commented Jul 8, 2020 •

edited