feat: added node groups API protos (#495)

* chore: update to gapic-generator-python 1.5.0

feat: add support for `google.cloud.<api>.__version__`
PiperOrigin-RevId: 484665853

Source-Link: googleapis/googleapis@8eb249a

Source-Link: googleapis/googleapis-gen@c8aa327
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzhhYTMyN2I1ZjQ3ODg2NWZjM2ZkOTFlM2MyNzY4ZTU0ZTI2YWQ0NCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update version in gapic_version.py

* add .release-please-manifest.json with correct version

* set manifest to true in .github/release-please.yml

* add release-please-config.json

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* chore: Update to gapic-generator-python 1.6.0

feat(python): Add typing to proto.Message based class attributes

feat(python): Snippetgen handling of repeated enum field

PiperOrigin-RevId: 487326846

Source-Link: googleapis/googleapis@da380c7

Source-Link: googleapis/googleapis-gen@61ef576
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNjFlZjU3NjJlZTY3MzFhMGNiYmZlYTIyZmQwZWVjZWU1MWFiMWM4ZSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* feat: new APIs added to reflect updates to the filestore service

- Add ENTERPRISE Tier
- Add snapshot APIs: RevertInstance, ListSnapshots, CreateSnapshot, DeleteSnapshot, UpdateSnapshot
- Add multi-share APIs: ListShares, GetShare, CreateShare, DeleteShare, UpdateShare
- Add ConnectMode to NetworkConfig (for Private Service Access support)
- New status codes (SUSPENDED/SUSPENDING, REVERTING/RESUMING)
- Add SuspensionReason (for KMS related suspension)
- Add new fields to Instance information: max_capacity_gb, capacity_step_size_gb, max_share_count, capacity_gb, multi_share_enabled

PiperOrigin-RevId: 487492758

Source-Link: googleapis/googleapis@5be5981

Source-Link: googleapis/googleapis-gen@ab0e217
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYWIwZTIxN2Y1NjBjYzJjMWFmYzExNDQxYzJlYWI2YjY5NTBlZmQyYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update path to snippet metadata json

* chore: Update gapic-generator-python to v1.6.1

PiperOrigin-RevId: 488036204

Source-Link: googleapis/googleapis@08f275f

Source-Link: googleapis/googleapis-gen@555c094
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTU1YzA5NDVlNjA2NDllMzg3MzlhZTY0YmM0NTcxOWNkZjcyMTc4ZiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fix(deps): Require google-api-core >=1.34.0, >=2.11.0

fix: Drop usage of pkg_resources

fix: Fix timeout default values

docs(samples): Snippetgen should call await on the operation coroutine before calling result

PiperOrigin-RevId: 493260409

Source-Link: googleapis/googleapis@fea4387

Source-Link: googleapis/googleapis-gen@387b734
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzg3YjczNDRjNzUyOWVlNDRiZTg0ZTYxM2IxOWE4MjA1MDhjNjEyYiJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* add gapic_version.py

* feat: added node groups API protos

PiperOrigin-RevId: 494840237

Source-Link: googleapis/googleapis@28449ec

Source-Link: googleapis/googleapis-gen@786b188
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNzg2YjE4ODljMDdjNGVhZDE2MDkzZDM3NWM1OTQ1NWEwNTc2ZWUyNSJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* update version in gapic_version.py

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* configure release please to use manifest

* remove obsolete entry in release please config

* use templated owlbot.py and setup.py

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* remove obsolete code in owlbot.py; restore noxfile

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>

.coveragerc

@@ -10,8 +10,3 @@ exclude_lines =
     pragma: NO COVER
     # Ignore debug-only repr
     def __repr__
-    # Ignore pkg_resources exceptions.
-    # This is added at the module level as a safeguard for if someone
-    # generates the code and tries to run it without pip installing. This
-    # makes it virtually impossible to test properly.
-    except pkg_resources.DistributionNotFound

.github/release-please.yml

@@ -1,5 +1,6 @@
 releaseType: python
 handleGHRelease: true
+manifest: true
 # NOTE: this section is generated by synthtool.languages.python
 # See https://github.com/googleapis/synthtool/blob/master/synthtool/languages/python.py
 branches:

.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "5.0.3"
+}

docs/dataproc_v1/node_group_controller.rst

@@ -0,0 +1,6 @@
+NodeGroupController
+-------------------------------------
+
+.. automodule:: google.cloud.dataproc_v1.services.node_group_controller
+    :members:
+    :inherited-members:

docs/dataproc_v1/services.rst

@@ -7,4 +7,5 @@ Services for Google Cloud Dataproc v1 API
     batch_controller
     cluster_controller
     job_controller
+    node_group_controller
     workflow_template_service

docs/dataproc_v1/types.rst

@@ -3,5 +3,4 @@ Types for Google Cloud Dataproc v1 API
 
 .. automodule:: google.cloud.dataproc_v1.types
     :members:
-    :undoc-members:
     :show-inheritance:

docs/index.rst

@@ -2,30 +2,33 @@
 
 .. include:: multiprocessing.rst
 
+
 API Reference
 -------------
 .. toctree::
-   :maxdepth: 2
+    :maxdepth: 2
+
+    dataproc_v1/services
+    dataproc_v1/types
 
-   dataproc_v1/services
-   dataproc_v1/types
 
 Migration Guide
 ---------------
 
-See the guide below for instructions on migrating to the 2.x release of this library.
+See the guide below for instructions on migrating to the latest version.
 
 .. toctree::
-   :maxdepth: 2
+    :maxdepth: 2
+
+    UPGRADING
 
-   UPGRADING
 
 Changelog
 ---------
 
 For a list of all ``google-cloud-dataproc`` releases:
 
 .. toctree::
-   :maxdepth: 2
+    :maxdepth: 2
 
-   changelog
+    changelog

google/cloud/dataproc/__init__.py

@@ -13,6 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+from google.cloud.dataproc import gapic_version as package_version
+
+__version__ = package_version.__version__
+
 
 from google.cloud.dataproc_v1.services.autoscaling_policy_service.client import (
     AutoscalingPolicyServiceClient,
@@ -36,6 +40,12 @@
 from google.cloud.dataproc_v1.services.job_controller.async_client import (
     JobControllerAsyncClient,
 )
+from google.cloud.dataproc_v1.services.node_group_controller.client import (
+    NodeGroupControllerClient,
+)
+from google.cloud.dataproc_v1.services.node_group_controller.async_client import (
+    NodeGroupControllerAsyncClient,
+)
 from google.cloud.dataproc_v1.services.workflow_template_service.client import (
     WorkflowTemplateServiceClient,
 )
@@ -83,6 +93,7 @@
 from google.cloud.dataproc_v1.types.batches import SparkSqlBatch
 from google.cloud.dataproc_v1.types.clusters import AcceleratorConfig
 from google.cloud.dataproc_v1.types.clusters import AutoscalingConfig
+from google.cloud.dataproc_v1.types.clusters import AuxiliaryNodeGroup
 from google.cloud.dataproc_v1.types.clusters import AuxiliaryServicesConfig
 from google.cloud.dataproc_v1.types.clusters import Cluster
 from google.cloud.dataproc_v1.types.clusters import ClusterConfig
@@ -107,6 +118,7 @@
 from google.cloud.dataproc_v1.types.clusters import ListClustersResponse
 from google.cloud.dataproc_v1.types.clusters import ManagedGroupConfig
 from google.cloud.dataproc_v1.types.clusters import MetastoreConfig
+from google.cloud.dataproc_v1.types.clusters import NodeGroup
 from google.cloud.dataproc_v1.types.clusters import NodeGroupAffinity
 from google.cloud.dataproc_v1.types.clusters import NodeInitializationAction
 from google.cloud.dataproc_v1.types.clusters import ReservationAffinity
@@ -119,6 +131,7 @@
 from google.cloud.dataproc_v1.types.clusters import VirtualClusterConfig
 from google.cloud.dataproc_v1.types.jobs import CancelJobRequest
 from google.cloud.dataproc_v1.types.jobs import DeleteJobRequest
+from google.cloud.dataproc_v1.types.jobs import DriverSchedulingConfig
 from google.cloud.dataproc_v1.types.jobs import GetJobRequest
 from google.cloud.dataproc_v1.types.jobs import HadoopJob
 from google.cloud.dataproc_v1.types.jobs import HiveJob
@@ -141,9 +154,13 @@
 from google.cloud.dataproc_v1.types.jobs import SubmitJobRequest
 from google.cloud.dataproc_v1.types.jobs import UpdateJobRequest
 from google.cloud.dataproc_v1.types.jobs import YarnApplication
+from google.cloud.dataproc_v1.types.node_groups import CreateNodeGroupRequest
+from google.cloud.dataproc_v1.types.node_groups import GetNodeGroupRequest
+from google.cloud.dataproc_v1.types.node_groups import ResizeNodeGroupRequest
 from google.cloud.dataproc_v1.types.operations import BatchOperationMetadata
 from google.cloud.dataproc_v1.types.operations import ClusterOperationMetadata
 from google.cloud.dataproc_v1.types.operations import ClusterOperationStatus
+from google.cloud.dataproc_v1.types.operations import NodeGroupOperationMetadata
 from google.cloud.dataproc_v1.types.shared import EnvironmentConfig
 from google.cloud.dataproc_v1.types.shared import ExecutionConfig
 from google.cloud.dataproc_v1.types.shared import GkeClusterConfig
@@ -202,6 +219,8 @@
     "ClusterControllerAsyncClient",
     "JobControllerClient",
     "JobControllerAsyncClient",
+    "NodeGroupControllerClient",
+    "NodeGroupControllerAsyncClient",
     "WorkflowTemplateServiceClient",
     "WorkflowTemplateServiceAsyncClient",
     "AutoscalingPolicy",
@@ -226,6 +245,7 @@
     "SparkSqlBatch",
     "AcceleratorConfig",
     "AutoscalingConfig",
+    "AuxiliaryNodeGroup",
     "AuxiliaryServicesConfig",
     "Cluster",
     "ClusterConfig",
@@ -250,6 +270,7 @@
     "ListClustersResponse",
     "ManagedGroupConfig",
     "MetastoreConfig",
+    "NodeGroup",
     "NodeGroupAffinity",
     "NodeInitializationAction",
     "ReservationAffinity",
@@ -262,6 +283,7 @@
     "VirtualClusterConfig",
     "CancelJobRequest",
     "DeleteJobRequest",
+    "DriverSchedulingConfig",
     "GetJobRequest",
     "HadoopJob",
     "HiveJob",
@@ -284,9 +306,13 @@
     "SubmitJobRequest",
     "UpdateJobRequest",
     "YarnApplication",
+    "CreateNodeGroupRequest",
+    "GetNodeGroupRequest",
+    "ResizeNodeGroupRequest",
     "BatchOperationMetadata",
     "ClusterOperationMetadata",
     "ClusterOperationStatus",
+    "NodeGroupOperationMetadata",
     "EnvironmentConfig",
     "ExecutionConfig",
     "GkeClusterConfig",

google/cloud/dataproc/gapic_version.py

@@ -0,0 +1,16 @@
+# -*- coding: utf-8 -*-
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+__version__ = "5.0.3"  # {x-release-please-version}

google/cloud/dataproc_v1/__init__.py

@@ -13,6 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+from google.cloud.dataproc import gapic_version as package_version
+
+__version__ = package_version.__version__
+
 
 from .services.autoscaling_policy_service import AutoscalingPolicyServiceClient
 from .services.autoscaling_policy_service import AutoscalingPolicyServiceAsyncClient
@@ -22,6 +26,8 @@
 from .services.cluster_controller import ClusterControllerAsyncClient
 from .services.job_controller import JobControllerClient
 from .services.job_controller import JobControllerAsyncClient
+from .services.node_group_controller import NodeGroupControllerClient
+from .services.node_group_controller import NodeGroupControllerAsyncClient
 from .services.workflow_template_service import WorkflowTemplateServiceClient
 from .services.workflow_template_service import WorkflowTemplateServiceAsyncClient
 
@@ -47,6 +53,7 @@
 from .types.batches import SparkSqlBatch
 from .types.clusters import AcceleratorConfig
 from .types.clusters import AutoscalingConfig
+from .types.clusters import AuxiliaryNodeGroup
 from .types.clusters import AuxiliaryServicesConfig
 from .types.clusters import Cluster
 from .types.clusters import ClusterConfig
@@ -71,6 +78,7 @@
 from .types.clusters import ListClustersResponse
 from .types.clusters import ManagedGroupConfig
 from .types.clusters import MetastoreConfig
+from .types.clusters import NodeGroup
 from .types.clusters import NodeGroupAffinity
 from .types.clusters import NodeInitializationAction
 from .types.clusters import ReservationAffinity
@@ -83,6 +91,7 @@
 from .types.clusters import VirtualClusterConfig
 from .types.jobs import CancelJobRequest
 from .types.jobs import DeleteJobRequest
+from .types.jobs import DriverSchedulingConfig
 from .types.jobs import GetJobRequest
 from .types.jobs import HadoopJob
 from .types.jobs import HiveJob
@@ -105,9 +114,13 @@
 from .types.jobs import SubmitJobRequest
 from .types.jobs import UpdateJobRequest
 from .types.jobs import YarnApplication
+from .types.node_groups import CreateNodeGroupRequest
+from .types.node_groups import GetNodeGroupRequest
+from .types.node_groups import ResizeNodeGroupRequest
 from .types.operations import BatchOperationMetadata
 from .types.operations import ClusterOperationMetadata
 from .types.operations import ClusterOperationStatus
+from .types.operations import NodeGroupOperationMetadata
 from .types.shared import EnvironmentConfig
 from .types.shared import ExecutionConfig
 from .types.shared import GkeClusterConfig
@@ -148,11 +161,13 @@
     "BatchControllerAsyncClient",
     "ClusterControllerAsyncClient",
     "JobControllerAsyncClient",
+    "NodeGroupControllerAsyncClient",
     "WorkflowTemplateServiceAsyncClient",
     "AcceleratorConfig",
     "AutoscalingConfig",
     "AutoscalingPolicy",
     "AutoscalingPolicyServiceClient",
+    "AuxiliaryNodeGroup",
     "AuxiliaryServicesConfig",
     "BasicAutoscalingAlgorithm",
     "BasicYarnAutoscalingConfig",
@@ -174,6 +189,7 @@
     "CreateAutoscalingPolicyRequest",
     "CreateBatchRequest",
     "CreateClusterRequest",
+    "CreateNodeGroupRequest",
     "CreateWorkflowTemplateRequest",
     "DataprocMetricConfig",
     "DeleteAutoscalingPolicyRequest",
@@ -184,6 +200,7 @@
     "DiagnoseClusterRequest",
     "DiagnoseClusterResults",
     "DiskConfig",
+    "DriverSchedulingConfig",
     "EncryptionConfig",
     "EndpointConfig",
     "EnvironmentConfig",
@@ -194,6 +211,7 @@
     "GetBatchRequest",
     "GetClusterRequest",
     "GetJobRequest",
+    "GetNodeGroupRequest",
     "GetWorkflowTemplateRequest",
     "GkeClusterConfig",
     "GkeNodePoolConfig",
@@ -230,7 +248,10 @@
     "ManagedCluster",
     "ManagedGroupConfig",
     "MetastoreConfig",
+    "NodeGroup",
     "NodeGroupAffinity",
+    "NodeGroupControllerClient",
+    "NodeGroupOperationMetadata",
     "NodeInitializationAction",
     "OrderedJob",
     "ParameterValidation",
@@ -242,6 +263,7 @@
     "QueryList",
     "RegexValidation",
     "ReservationAffinity",
+    "ResizeNodeGroupRequest",
     "RuntimeConfig",
     "RuntimeInfo",
     "SecurityConfig",

google/cloud/dataproc_v1/gapic_metadata.json

@@ -301,6 +301,50 @@
         }
       }
     },
+    "NodeGroupController": {
+      "clients": {
+        "grpc": {
+          "libraryClient": "NodeGroupControllerClient",
+          "rpcs": {
+            "CreateNodeGroup": {
+              "methods": [
+                "create_node_group"
+              ]
+            },
+            "GetNodeGroup": {
+              "methods": [
+                "get_node_group"
+              ]
+            },
+            "ResizeNodeGroup": {
+              "methods": [
+                "resize_node_group"
+              ]
+            }
+          }
+        },
+        "grpc-async": {
+          "libraryClient": "NodeGroupControllerAsyncClient",
+          "rpcs": {
+            "CreateNodeGroup": {
+              "methods": [
+                "create_node_group"
+              ]
+            },
+            "GetNodeGroup": {
+              "methods": [
+                "get_node_group"
+              ]
+            },
+            "ResizeNodeGroup": {
+              "methods": [
+                "resize_node_group"
+              ]
+            }
+          }
+        }
+      }
+    },
     "WorkflowTemplateService": {
       "clients": {
         "grpc": {

google/cloud/dataproc_v1/gapic_version.py

@@ -0,0 +1,16 @@
+# -*- coding: utf-8 -*-
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+__version__ = "5.0.3"  # {x-release-please-version}