AMBARI-22945. Enhance host components API to support multiple host component instances. #477

swapanshridhar · 2018-02-27T08:07:07Z

What changes were proposed in this pull request?

BACKGROUND:

Given that in Ambari, we refer components of a services in 2 ways:

1. One from the Service component APIs :
http://:8080/api/v1/clusters//services//components/
2. From the host component APIs (which tell us the host on which the current component is resides)
http://:8080/api/v1/clusters//hosts//host_components/

Note that, we are referring both 1and 2 using the component names

But in the multi-component world for Ambari, we will come up with a situation like this, as shown below:

A. If core Service Group has its HDFS_CLIENT type component named as HDFS_CLIENT
Core -> HDFS -> HDFS_CLIENT
B. And at the same time, we have edw Service Group having its HDFS_CLIENT type component named as HDFS_CLIENT :
edw -> HDFS -> HDFS_CLIENT

There is no good way to distinguish a given host component in A and B, if we continue to refer them with the component name end point (No. 2 API call).
However, No. 1 call is still fine as we will not allow 2 names for the same component within a given service, thus making it unique and allowing us to continue using the component names endpoint.

WORK DONE

In order to support multi-instance for components of a given service, we need to enhance the Host Components API so that they can distinguish one component instance of a service from another component instance of a service with the same Name.

The way to achieve this is to move Host Components API to be ID (number) based end point, compared to earlier being a name based endpoint. This will allow us to distinguish Core -> HDFS -> HDFS_CLIENT from edw -> HDFS -> HDFS_CLIENT, as an example.

Thus, following changes are required:

New field : component_type (eg: HDFS_CLIENT, HIVE_SERVER, ZOOKEEPER_SERVER etc)
Existing field : component_name will now hold the actual name given for the component at the time of install. For example:
- HDFS_CLIENT_EDW for component_type HDFS_CLIENT
- HIVE_SERVER FOR component_type HIVE_SERVER (name can be same as type also)
- HIVE_SERVER1 for component_type HIVE_SERVER
- ZOOKEEPER_SERVER_FOR_KAFKA for component_type ZOOKEEPER_SERVER, and so forth.
A way to identify the HOST component uniquely via API calls, by way of referring them by ID instead of name.

MODIFIED API CALLs:

POST SERVICE COMPONENT API call

CHANGE : New field component_type
POST http://{{AmbariServer}}:8080/api/v1/clusters//servicegroups/core/services/HDFS/components

Body :

 [{"ServiceComponentInfo":{"component_name":"HDFS_CLIENT2","component_type":"HDFS_CLIENT"}}]

Response:

{
  "resources" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
      "ServiceComponentInfo" : {
        "category" : null,
        "cluster_id" : 2,
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "component_type" : "HDFS_CLIENT",
        "desired_stack" : "HDP-2.6",
        "desired_version" : "NOT_REQUIRED",
        "display_name" : "HDFS Client",
        "id" : 51,
        "recovery_enabled" : false,
        "service_group_id" : 2,
        "service_group_name" : "core",
        "service_id" : 3,
        "service_name" : "HDFS",
        "service_type" : "HDFS",
        "state" : "STARTED",
        "total_count" : {
          "installFailedCount" : 0,
          "unknownCount" : 0,
          "installedCount" : 0,
          "initCount" : 0,
          "installedAndMaintenanceOffCount" : 0,
          "startedCount" : 0,
          "totalCount" : 0
        }
      }
    }
  ]
}

POST HOST COMPONENT API call

CHANGE : New field component_type
POST http://{{AmbariServer}}:8080/api/v1/clusters//hosts//host_components
Body

{"HostRoles":{ "component_name":"HDFS_CLIENT2", "component_type":"HDFS_CLIENT", "service_group_name": "core", "service_name": "HDFS"}}

Response:

   {
  "resources" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
      "HostRoles" : {
        "actual_configs" : { },
        "cluster_id" : 2,
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "component_type" : "HDFS_CLIENT",
        "desired_admin_state" : null,
        "desired_repository_version" : "2.6.4.0-91",
        "desired_stack_id" : "HDP-2.6",
        "display_name" : "HDFS_CLIENT2",
        "host_name" : "<host1>",
        "id" : 51,
        "maintenance_state" : null,
        "public_host_name" : "<host1>",
        "service_group_id" : 2,
        "service_group_name" : "core",
        "service_id" : 3,
        "service_name" : "HDFS",
        "service_type" : "HDFS",
        "stale_configs" : false,
        "state" : "INIT",
        "upgrade_state" : "NONE",
        "version" : "UNKNOWN"
      },
      "host" : {
        "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>"
      }
    }
  ]
}

GET SERVICE COMPONENT:
- CHANGE: host_components sub-resource will be referenced with end point as ID one.
- GET http://{{AmbariServer}}:8080/api/v1/clusters//servicegroups/core/services/HDFS/components/HDFS_CLIENT2
- Response:

   {
  "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
  "ServiceComponentInfo" : {
    "category" : null,
    "cluster_id" : 2,
    "cluster_name" : "c1",
    "component_name" : "HDFS_CLIENT2",
    "component_type" : "HDFS_CLIENT",
    "desired_stack" : "HDP-2.6",
    "desired_version" : "2.6.4.0-91",
    "display_name" : "HDFS Client",
    "id" : 51,
    "init_count" : 1,
    "install_failed_count" : 0,
    "installed_and_maintenance_off_count" : 0,
    "installed_count" : 0,
    "recovery_enabled" : "false",
    "repository_state" : "NOT_REQUIRED",
    "service_group_id" : 2,
    "service_group_name" : "core",
    "service_id" : 3,
    "service_name" : "HDFS",
    "service_type" : "HDFS",
    "started_count" : 0,
    "state" : "STARTED",
    "total_count" : 1,
    "unknown_count" : 0
  },
  "host_components" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
      "HostRoles" : {
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "host_name" : "<host1>",
        "id" : 51,
        "service_group_name" : "core",
        "service_name" : "HDFS"
      }
    }
  ]
}

GET HOST COMPONENT:
- CHANGE: Called with ID based end point
- GET http://{{AmbariServer}}:8080/api/v1/clusters//hosts//host_components/51
- Response:

   {
  "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
  "HostRoles" : {
    "cluster_name" : "c1",
    "component_name" : "HDFS_CLIENT2",
    "component_type" : "HDFS_CLIENT",
    "desired_repository_version" : "2.6.4.0-91",
    "desired_stack_id" : "HDP-2.6",
    "desired_state" : "INIT",
    "display_name" : "HDFS_CLIENT2",
    "host_name" : "<host1>",
    "id" : 51,
    "maintenance_state" : "OFF",
    "public_host_name" : "<host1>",
    "reload_configs" : false,
    "service_group_name" : "core",
    "service_name" : "HDFS",
    "stale_configs" : false,
    "state" : "INIT",
    "upgrade_state" : "NONE",
    "version" : "UNKNOWN",
    "actual_configs" : { }
  },
  "host" : {
    "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>"
  },
  "processes" : [ ],
  "component" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
      "ServiceComponentInfo" : {
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "service_group_name" : "core",
        "service_name" : "HDFS"
      }
    }
  ]
}

UPDATE SERVICE COMPONENTS
- NO CHANGE
UPDATE HOST COMPONENTS
- CHANGE: API end point is ID based now.
DELETE SERVICE COMPONENT:
- No CHANGE
DELETE HOST COMPONENT:
- CHANGE: API end point is ID based now.

How was this patch tested?

Tested on live cluster, Details given above.

asfgit · 2018-02-27T10:08:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/835/
Test FAILed.
Test FAILured.

mradha25 · 2018-02-27T19:08:14Z

ambari-server/src/main/java/org/apache/ambari/server/api/services/HostComponentService.java

  @Produces("text/plain")
  public Response createHostComponent(String body, @Context HttpHeaders headers, @Context UriInfo ui,
-                                   @PathParam("hostComponentName") String hostComponentName) {
+                                   @PathParam("hostComponentId") String hostComponentId) {


Hi,
For POST call we will not have hostComponentId in the PathParam since it has not been created yet.

Removed this POST call.

mradha25 · 2018-02-27T20:07:01Z

...rc/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java

@@ -677,6 +684,8 @@ protected RequestStageContainer updateHostComponents(RequestStageContainer stage
            + ", clusterId=" + cluster.getClusterId()
            + ", serviceName=" + sch.getServiceName()
            + ", componentName=" + sch.getServiceComponentName()
+            + ", componentType=" + sch.getServiceComponentType()
+            + ", componentType=" + sch.getServiceComponentType()


Duplicate statement

mradha25 · 2018-02-27T20:17:03Z

...rc/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java

+    ServiceComponentHostRequest serviceComponentHostRequest = null;
+    if (properties.get(HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID) != null) {
+      Long hostComponentId = properties.get(HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID) instanceof String ?
+              Long.parseLong((String) properties.get(HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID)) :


Here we are checking if its an instance of String, and then casting it to String and then using parseLong. If we have already checked that its an instance of String then we should be able to simply do
Long.parseLong(properties.get(HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID)).
Might be simpler to just use :
Long hostComponentId = Long.valueOf((String) properties.get(HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID)
parseLong returns a primitive type so you can choose to use valueOf to get an object of Long type.

mradha25 · 2018-02-27T20:29:22Z

...i-server/src/main/java/org/apache/ambari/server/orm/dao/ServiceComponentDesiredStateDAO.java

+    query.setParameter("clusterId", clusterId);
+    query.setParameter("serviceGroupId", serviceGroupId);
+    query.setParameter("serviceId", serviceId);
+    query.setParameter("id", componentId);


Do we need clusterId, serviceGroupId, serviceId in this case?

mradha25 · 2018-02-27T20:37:56Z

.../src/main/java/org/apache/ambari/server/orm/entities/ServiceComponentDesiredStateEntity.java

+      "AND scds.componentName = :componentName " +
+      "AND scds.componentType = :componentType" ),
+  @NamedQuery(
+    name = "ServiceComponentDesiredStateEntity.findById",


Id will already be unique. Do we need the clusterId, serviceGroupId and serviceId?

mradha25 · 2018-02-27T20:49:37Z

ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java

+  public String getComponentName(Long componentId) throws AmbariException {
+    for (Service service : services.values()) {
+      for (ServiceComponent component : service.getServiceComponents().values()) {
+        if (component.getId() == componentId) {


use .equals

mradha25 · 2018-02-27T20:50:25Z

ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java

+  public String getComponentType(Long componentId) throws AmbariException {
+    for (Service service : services.values()) {
+      for (ServiceComponent component : service.getServiceComponents().values()) {
+        if (component.getId() == componentId) {


use .equals

benyoka · 2018-02-28T11:01:59Z

ambari-server/src/main/java/org/apache/ambari/server/ServiceComponentNotFoundException.java

+            + ", serviceType=" + serviceType
+            + ", serviceComponentId=" + serviceComponentId);
+  }
+
 }


Where is serviceComponentId stored in the DB?

It should come from servicecomponentdesiredstate table.

swapanshridhar · 2018-03-02T10:54:42Z

AMBARI-22945. Enhance host components API to support multiple host co… …

Adds/changes the following:

Fixes the failing UTs
Cleans up the code changes made around AMS controller code. We will be as of now continue to pass String value as component Name. Ams code needs to be enhanced to uniquely identify a given component instance using SG name, Service Name and Component Name, as they can't use id.
UI code changes to pass in service_group_name and service_name while doing POST for HOST Component(s).

Manual testing done as follows:

Installed cluster for HDFS and ZK with new bits. Works.
Tested Host API bulk operation, details here : https://issues.apache.org/jira/browse/AMBARI-22945?focusedCommentId=16383438&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16383438

swapanshridhar · 2018-03-02T11:16:05Z

AMBARI-22945. Enhance host components API to support multiple host co…

Updated ''AmbariManagementControllerTest.java' after merge.

asfgit · 2018-03-02T13:55:49Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/925/
Test FAILed.
Test FAILured.

asfgit · 2018-03-02T15:52:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/927/
Test FAILed.
Test FAILured.

swapanshridhar · 2018-03-02T19:15:59Z

@avijayanhwx Can you review this change for Metrics related bare minimum changes that I have done.
You will have to build your further changes around it. I have put tag :

// TODO : Multi_Metrics_Changes.

where I have identified further code changes are required to uniquely identify one components from another.

asfgit · 2018-03-05T03:25:40Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/954/
Test FAILed.
Test FAILured.

asfgit · 2018-03-05T12:04:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/957/
Test FAILed.
Test FAILured.

avijayanhwx

Metrics changes are OK for a preliminary patch. After implementing support for multi instance metrics in AMS, we can use a follow up JIRA to clean up metrics related integration points in Ambari Server.

avijayanhwx · 2018-03-05T19:30:48Z

...rc/main/java/org/apache/ambari/server/controller/metrics/MetricsCollectorHAClusterState.java

+    try {
+      componentId = managementController.getClusters().getCluster(clusterName).getComponentId(componentName);
+    } catch (AmbariException e) {
+      e.printStackTrace();


Can we log the exception instead of printStackTrace ?

mradha25 · 2018-03-05T19:52:04Z

...ri-server/src/main/java/org/apache/ambari/server/api/services/RootClusterSettingService.java

@@ -70,7 +70,7 @@ public RootClusterSettingService() {
          response = ReadOnlyConfigurationResponse.ReadOnlyConfigurationResponseSwagger.class, responseContainer = RESPONSE_CONTAINER_LIST)
  @ApiImplicitParams({
          @ApiImplicitParam(name = QUERY_FIELDS, value = QUERY_FILTER_DESCRIPTION, dataType = DATA_TYPE_STRING,
-                  paramType = PARAM_TYPE_QUERY, defaultValue = MpackResourceProvider.MPACK_RESOURCE_ID),
+                  paramType = PARAM_TYPE_QUERY, defaultValue = MpackResourceProvider.MPACK_ID),


You can remove this line completely.

…mponent instances.

adoroszlai · 2018-03-05T22:34:18Z

...rc/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java

+              (String) properties.get(HOST_COMPONENT_HOST_NAME_PROPERTY_ID),
+              (String) properties.get(HOST_COMPONENT_DESIRED_STATE_PROPERTY_ID));
+    }
+
    serviceComponentHostRequest.setState((String) properties.get(HOST_COMPONENT_STATE_PROPERTY_ID));


This results in NPE if HOST_COMPONENT_HOST_COMPONENT_ID_PROPERTY_ID is not present.

Fixed in #566 and AMBARI-23158

asfgit · 2018-03-06T08:38:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/984/
Test FAILed.
Test FAILured.

swapanshridhar assigned d0zen1, jayush, adoroszlai, mradha25 and vbrodetskyi Feb 27, 2018

swapanshridhar mentioned this pull request Feb 27, 2018

AMBARI-22945. Enhance host components API to support multiple host component instances. #467

Closed

adoroszlai requested review from d0zen1, jayush, vbrodetskyi, adoroszlai, benyoka and mradha25 February 27, 2018 11:49

adoroszlai assigned swapanshridhar and unassigned d0zen1, jayush, vbrodetskyi, adoroszlai and mradha25 Feb 27, 2018

adoroszlai added the API label Feb 27, 2018

mradha25 suggested changes Feb 27, 2018

View reviewed changes

benyoka reviewed Feb 28, 2018

View reviewed changes

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from d6fc1cd to df4aa4b Compare March 2, 2018 09:33

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from df4aa4b to 93d949c Compare March 2, 2018 11:09

swapanshridhar requested a review from avijayanhwx March 2, 2018 19:11

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from 93d949c to 2d7d560 Compare March 5, 2018 02:49

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from 2d7d560 to 34080a3 Compare March 5, 2018 09:38

jayush approved these changes Mar 5, 2018

View reviewed changes

avijayanhwx approved these changes Mar 5, 2018

View reviewed changes

mradha25 approved these changes Mar 5, 2018

View reviewed changes

AMBARI-22945. Enhance host components API to support multiple host co…

2c844e8

…mponent instances.

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from 34080a3 to 2c844e8 Compare March 5, 2018 22:18

swapanshridhar merged commit 49fa0ac into apache:branch-feature-AMBARI-14714 Mar 5, 2018

adoroszlai reviewed Mar 5, 2018

View reviewed changes

swapanshridhar deleted the AMBARI-22945-branch-feature-AMBARI-14714 branch March 6, 2018 20:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AMBARI-22945. Enhance host components API to support multiple host component instances. #477

AMBARI-22945. Enhance host components API to support multiple host component instances. #477

swapanshridhar commented Feb 27, 2018 •

edited

asfgit commented Feb 27, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

mradha25 Feb 27, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

mradha25 Feb 27, 2018

swapanshridhar Feb 28, 2018

benyoka Feb 28, 2018

swapanshridhar Mar 2, 2018

swapanshridhar commented Mar 2, 2018 •

edited

swapanshridhar commented Mar 2, 2018

asfgit commented Mar 2, 2018

asfgit commented Mar 2, 2018

swapanshridhar commented Mar 2, 2018 •

edited

asfgit commented Mar 5, 2018

asfgit commented Mar 5, 2018

avijayanhwx left a comment

avijayanhwx Mar 5, 2018

mradha25 Mar 5, 2018

adoroszlai Mar 5, 2018

swapanshridhar Mar 6, 2018

asfgit commented Mar 6, 2018

AMBARI-22945. Enhance host components API to support multiple host component instances. #477

AMBARI-22945. Enhance host components API to support multiple host component instances. #477

Conversation

swapanshridhar commented Feb 27, 2018 • edited

What changes were proposed in this pull request?

How was this patch tested?

asfgit commented Feb 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swapanshridhar commented Mar 2, 2018 • edited

swapanshridhar commented Mar 2, 2018

asfgit commented Mar 2, 2018

asfgit commented Mar 2, 2018

swapanshridhar commented Mar 2, 2018 • edited

asfgit commented Mar 5, 2018

asfgit commented Mar 5, 2018

avijayanhwx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfgit commented Mar 6, 2018

swapanshridhar commented Feb 27, 2018 •

edited

swapanshridhar commented Mar 2, 2018 •

edited

swapanshridhar commented Mar 2, 2018 •

edited