AMBARI-22945. Enhance host components API to support multiple host component instances. #467

swapanshridhar · 2018-02-26T06:22:31Z

What changes were proposed in this pull request?

BACKGROUND:

Given that in Ambari, we refer components of a services in 2 ways:

1. One from the Service component APIs :
http://:8080/api/v1/clusters//services//components/
2. From the host component APIs (which tell us the host on which the current component is resides)
http://:8080/api/v1/clusters//hosts//host_components/

Note that, we of now, we are referring both 1and 2 using the component names

But in the multi-component world for Ambari, we will come up with a situation like this, as shown below:

A. If core Service Group has its HDFS_CLIENT type component named as HDFS_CLIENT
Core -> HDFS -> HDFS_CLIENT
B. And at the same time, we have edw Service Group having its HDFS_CLIENT type component named as HDFS_CLIENT :
edw -> HDFS -> HDFS_CLIENT

There is no good way to distinguish a given host component in A and B, if we continue to refer them with the component name end point (No. 2 API call).
However, No. 1 call is still fine as we will not allow 2 names for the same component within a given service, thus making it unique and allowing us to continue using the component names endpoint.

WORK DONE

In order to support multi-instance for components of a given service, we need to enhance the Host Components API so that they can distinguish one component instance of a service from another component instance of a service with the same Name.

The way to achieve this is to move Host Components API to be ID (number) based end point, compared to earlier being a name based endpoint. This will allow us to distinguish Core -> HDFS -> HDFS_CLIENT from edw -> HDFS -> HDFS_CLIENT, as an example.

Thus, following changes are required:

New field : component_type (eg: HDFS_CLIENT, HIVE_SERVER, ZOOKEEPER_SERVER etc)
Existing field : component_name will now hold the actual name given for the component at the time of install. For example:
- HDFS_CLIENT_EDW for component_type HDFS_CLIENT
- HIVE_SERVER FOR component_type HIVE_SERVER (name can be same as type also)
- HIVE_SERVER1 for component_type HIVE_SERVER
- ZOOKEEPER_SERVER_FOR_KAFKA for component_type ZOOKEEPER_SERVER, and so forth.
A way to identify the HOST component uniquely via API calls, by way of referring them by ID instead of name.

MODIFIED API CALLs:

POST SERVICE COMPONENT API call

CHANGE : New field component_type
POST http://{{AmbariServer}}:8080/api/v1/clusters//servicegroups/core/services/HDFS/components

Body :

 [{"ServiceComponentInfo":{"component_name":"HDFS_CLIENT2","component_type":"HDFS_CLIENT"}}]

Response:

{
  "resources" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
      "ServiceComponentInfo" : {
        "category" : null,
        "cluster_id" : 2,
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "component_type" : "HDFS_CLIENT",
        "desired_stack" : "HDP-2.6",
        "desired_version" : "NOT_REQUIRED",
        "display_name" : "HDFS Client",
        "id" : 51,
        "recovery_enabled" : false,
        "service_group_id" : 2,
        "service_group_name" : "core",
        "service_id" : 3,
        "service_name" : "HDFS",
        "service_type" : "HDFS",
        "state" : "STARTED",
        "total_count" : {
          "installFailedCount" : 0,
          "unknownCount" : 0,
          "installedCount" : 0,
          "initCount" : 0,
          "installedAndMaintenanceOffCount" : 0,
          "startedCount" : 0,
          "totalCount" : 0
        }
      }
    }
  ]
}

POST HOST COMPONENT API call

CHANGE : New field component_type
POST http://{{AmbariServer}}:8080/api/v1/clusters//hosts//host_components
Body

{"HostRoles":{"cluster_name": "c1","component_name":"HDFS_CLIENT2","component_type":"HDFS_CLIENT"}}

Response:

   {
  "resources" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
      "HostRoles" : {
        "actual_configs" : { },
        "cluster_id" : 2,
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "component_type" : "HDFS_CLIENT",
        "desired_admin_state" : null,
        "desired_repository_version" : "2.6.4.0-91",
        "desired_stack_id" : "HDP-2.6",
        "display_name" : "HDFS_CLIENT2",
        "host_name" : "<host1>",
        "id" : 51,
        "maintenance_state" : null,
        "public_host_name" : "<host1>",
        "service_group_id" : 2,
        "service_group_name" : "core",
        "service_id" : 3,
        "service_name" : "HDFS",
        "service_type" : "HDFS",
        "stale_configs" : false,
        "state" : "INIT",
        "upgrade_state" : "NONE",
        "version" : "UNKNOWN"
      },
      "host" : {
        "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>"
      }
    }
  ]
}

GET SERVICE COMPONENT:
- CHANGE: host_components sub-resource will be referenced with end point as ID one.
- GET http://{{AmbariServer}}:8080/api/v1/clusters//servicegroups/core/services/HDFS/components/HDFS_CLIENT2
- Response:

   {
  "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
  "ServiceComponentInfo" : {
    "category" : null,
    "cluster_id" : 2,
    "cluster_name" : "c1",
    "component_name" : "HDFS_CLIENT2",
    "component_type" : "HDFS_CLIENT",
    "desired_stack" : "HDP-2.6",
    "desired_version" : "2.6.4.0-91",
    "display_name" : "HDFS Client",
    "id" : 51,
    "init_count" : 1,
    "install_failed_count" : 0,
    "installed_and_maintenance_off_count" : 0,
    "installed_count" : 0,
    "recovery_enabled" : "false",
    "repository_state" : "NOT_REQUIRED",
    "service_group_id" : 2,
    "service_group_name" : "core",
    "service_id" : 3,
    "service_name" : "HDFS",
    "service_type" : "HDFS",
    "started_count" : 0,
    "state" : "STARTED",
    "total_count" : 1,
    "unknown_count" : 0
  },
  "host_components" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
      "HostRoles" : {
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "host_name" : "<host1>",
        "id" : 51,
        "service_group_name" : "core",
        "service_name" : "HDFS"
      }
    }
  ]
}

GET HOST COMPONENT:
- CHANGE: Called with ID based end point
- GET http://{{AmbariServer}}:8080/api/v1/clusters//hosts//host_components/51
- Response:

   {
  "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>/host_components/51",
  "HostRoles" : {
    "cluster_name" : "c1",
    "component_name" : "HDFS_CLIENT2",
    "component_type" : "HDFS_CLIENT",
    "desired_repository_version" : "2.6.4.0-91",
    "desired_stack_id" : "HDP-2.6",
    "desired_state" : "INIT",
    "display_name" : "HDFS_CLIENT2",
    "host_name" : "<host1>",
    "id" : 51,
    "maintenance_state" : "OFF",
    "public_host_name" : "<host1>",
    "reload_configs" : false,
    "service_group_name" : "core",
    "service_name" : "HDFS",
    "stale_configs" : false,
    "state" : "INIT",
    "upgrade_state" : "NONE",
    "version" : "UNKNOWN",
    "actual_configs" : { }
  },
  "host" : {
    "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/hosts/<host1>"
  },
  "processes" : [ ],
  "component" : [
    {
      "href" : "http://{{AmbariServer}}:8080/api/v1/clusters/<clusterName>/servicegroups/core/services/HDFS/components/HDFS_CLIENT2",
      "ServiceComponentInfo" : {
        "cluster_name" : "c1",
        "component_name" : "HDFS_CLIENT2",
        "service_group_name" : "core",
        "service_name" : "HDFS"
      }
    }
  ]
}

UPDATE SERVICE COMPONENTS
- NO CHANGE
UPDATE HOST COMPONENTS
- CHANGE: API end point is ID based now.
DELETE SERVICE COMPONENT:
- No CHANGE
DELETE HOST COMPONENT:
- CHANGE: API end point is ID based now.

How was this patch tested?

Tested on live cluster, Details given above.

asfgit · 2018-02-26T08:50:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/813/
Test FAILed.
Test FAILured.

adoroszlai

I've run unit tests that were changed in this PR, and there are about 100 additional errors compared to the base feature branch.

before:

Tests run: 455, Failures: 11, Errors: 48, Skipped: 8

after:

Tests run: 455, Failures: 14, Errors: 148, Skipped: 8

swapanshridhar · 2018-02-26T17:51:37Z

I've run unit tests that were changed in this PR, and there are about 100 additional errors compared to the base feature branch.

before:

Tests run: 455, Failures: 11, Errors: 48, Skipped: 8
after:

Tests run: 455, Failures: 14, Errors: 148, Skipped: 8
Merge state

Hi @adoroszlai, I have as of now worked on getting the tests compiled, so that we get the changes in and unblock others.
At the same time, I have opened aJIRA to track the UT failures and fix them in shortly.
https://hortonworks.jira.com/browse/BUG-97407
CC @jayush

adoroszlai · 2018-02-26T18:55:37Z

@swapanshridhar It took only about 2 hours to fix unit tests broken by #234 in #473, including the time for the initial run of all unit tests. I think creating follow-up tasks, re-running the tests later, waiting for more Jenkins builds, etc. is more overhead than getting them right the first time.

Living with broken tests makes it harder for others to check if they are breaking anything, since it's not as simple as "passing" vs. "failing", rather n vs. n+m failing tests. You have to check the logs to notice the difference, which apparently most people skip.

Also, a test may be broken by one change, but if it is not fixed, then another change could be also breaking it without the author noticing.

…mponent instances.

swapanshridhar · 2018-02-27T06:39:26Z

AMBARI-22945. Enhance host components API to support multiple host co… …
…mponent instances.
a9d7fe4
Merge state

Following was changed in it :

HostComponentStateEntity and HostComponentDesiredStateEntity : findByIndex will take in only the primary key only to do the lookup.
Added a constraint of passing service_group_name and service_name while doing POST Host components.
For Host Component and Service Component API, passing of component_type as of now is made optional, just like we have it for Services.
Added a Unique requirement in DB and Entity for hostcomponentstate and removed host_id from Unique constraint for hostcomponentdesiredstate
Refactored the Metrics code where we query for component name and type.

asfgit · 2018-02-27T07:02:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/832/
Test FAILed.
Test FAILured.

swapanshridhar · 2018-02-27T07:18:11Z

Add files via upload
Removed Unused Imports.

swapanshridhar · 2018-02-27T07:21:15Z

@swapanshridhar It took only about 2 hours to fix unit tests broken by #234 in #473, including the time for the initial run of all unit tests. I think creating follow-up tasks, re-running the tests later, waiting for more Jenkins builds, etc. is more overhead than getting them right the first time.

Living with broken tests makes it harder for others to check if they are breaking anything, since it's not as simple as "passing" vs. "failing", rather n vs. n+m failing tests. You have to check the logs to notice the difference, which apparently most people skip.

Also, a test may be broken by one change, but if it is not fixed, then another change could be also breaking it without the author noticing.

Okay. As of now, I have worked on changes suggested by @jayush which I have mentioned already as part of updates to the Pull request. I'll work on getting the UT's fixed.

asfgit · 2018-02-27T07:23:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/833/
Test FAILed.
Test FAILured.

adoroszlai · 2018-02-27T07:25:05Z

@swapanshridhar Thanks for working on the UTs, too.

Can you please check your latest commit (f126f65)? It added a patch file instead of the actual changes.

asfgit · 2018-02-27T07:27:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/834/
Test FAILed.
Test FAILured.

swapanshridhar · 2018-02-27T08:11:01Z

Closing this Pull request. Tracking changes at : #477

swapanshridhar assigned d0zen1, jayush, vbrodetskyi, adoroszlai and mradha25 Feb 26, 2018

adoroszlai requested review from d0zen1, jayush, vbrodetskyi, adoroszlai and mradha25 February 26, 2018 07:27

adoroszlai assigned swapanshridhar and unassigned d0zen1, jayush, vbrodetskyi, adoroszlai and mradha25 Feb 26, 2018

adoroszlai suggested changes Feb 26, 2018

View reviewed changes

AMBARI-22945. Enhance host components API to support multiple host co…

a9d7fe4

…mponent instances.

swapanshridhar force-pushed the AMBARI-22945-branch-feature-AMBARI-14714 branch from 724d37f to a9d7fe4 Compare February 27, 2018 06:28

Add files via upload

f126f65

swapanshridhar closed this Feb 27, 2018

swapanshridhar reopened this Feb 27, 2018

swapanshridhar closed this Feb 27, 2018

swapanshridhar deleted the AMBARI-22945-branch-feature-AMBARI-14714 branch February 27, 2018 08:11

mariasnyk mentioned this pull request Nov 30, 2023

[Snyk] Security upgrade ember-cli-babel from 5.2.8 to 8.0.0 mariasnyk/ambari#101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AMBARI-22945. Enhance host components API to support multiple host component instances. #467

AMBARI-22945. Enhance host components API to support multiple host component instances. #467

swapanshridhar commented Feb 26, 2018 •

edited

asfgit commented Feb 26, 2018

adoroszlai left a comment

swapanshridhar commented Feb 26, 2018

adoroszlai commented Feb 26, 2018

swapanshridhar commented Feb 27, 2018

asfgit commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

asfgit commented Feb 27, 2018

adoroszlai commented Feb 27, 2018

asfgit commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

AMBARI-22945. Enhance host components API to support multiple host component instances. #467

AMBARI-22945. Enhance host components API to support multiple host component instances. #467

Conversation

swapanshridhar commented Feb 26, 2018 • edited

What changes were proposed in this pull request?

How was this patch tested?

asfgit commented Feb 26, 2018

adoroszlai left a comment

Choose a reason for hiding this comment

swapanshridhar commented Feb 26, 2018

adoroszlai commented Feb 26, 2018

swapanshridhar commented Feb 27, 2018

asfgit commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

asfgit commented Feb 27, 2018

adoroszlai commented Feb 27, 2018

asfgit commented Feb 27, 2018

swapanshridhar commented Feb 27, 2018

swapanshridhar commented Feb 26, 2018 •

edited