[RFE] Parallel Filestore deploy and destroy

[m4r1k]

Currently, when deploying a blueprint with two Filestore, the deployment happens one after the other.
This process is suboptimal because there is no technical reason to wait for the first Filestore to be ready before performing the next deployment.
Likewise, when destroying a cluster, Filestore resource deletion happens sequentially.

I'd like to propose improving this process and considerably shirking both deployment and deletion timelines.

WDYT?

[cboneti]

Hi Federico,
As far as we know, that is a filestore issue. I will investigate that and come back to you shortly.

[cboneti]

Hi Federico,

I created a simple blueprint to try to reproduce the error and, here, the instances seem to be created in parallel.

I also had the impression of seeing this sequential creation before, could it have been circumstantial (based on the number of things being already created)? Could you try again with -parallelism=20 during the terraform apply?

FYI, the yaml I used for test now was:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---

blueprint_name: filestore-test

vars:
  project_id:  ## Set GCP Project ID Here ##
  deployment_name: filestore-test
  region: us-central1
  zone: us-central1-c

deployment_groups:
- group: primary
  modules:
  - id: network1
    source: modules/network/vpc

  - id: homefs
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home

  - id: homefs2
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home2

  - id: homefs3
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home3

  - id: homefs4
    source: modules/file-system/filestore
    use: [network1]
    settings:
      local_mount: /home4

  - id: compute
    source: modules/compute/vm-instance
    kind: terraform
    use: [network1]
    settings:
      name_prefix: vm
      instance_count: 1

And the output seems to be parallel.

What do you think?

[m4r1k]

Hey Carlos,

The output is definitely in parallel but on GCP the Filestores go up one at the time.

The following test has just been done on 1.4.1

[nick-stroud]

We are still working to track down if the observed behavior is a result of the terraform provider implementation or google cloud implementation.

[cboneti]

I found the root cause of this serialization to be described here: hashicorp/terraform-provider-google#9007
I am pursuing both the terraform and the internal Filestore team to see if we can change back the implementation in the provider and fix the underlying issue. This will take time.

[cboneti]

GoogleCloudPlatform/magic-modules#6672 should address the issue.

[tpdownes]

v4.41.0
of the Terraform provider for Google Cloud removes the constraint for serial creation of Filestore instances. Any new blueprint deployment should automatically get this release of the provider. You can upgrade an existing environment by running terraform init -upgrade in an existing Terraform folder.