REST api on Selenium Hub by krmahadevan · Pull Request #3460 · SeleniumHQ/selenium

krmahadevan · 2017-02-04T08:42:05Z

By placing an X in the preceding checkbox, I verify that I have signed the Contributor License Agreement

krmahadevan · 2017-02-04T08:44:26Z

This PR represents the original work that was started off by @dima-groupon as part of #265 and later rebased and revived by @mach6

krmahadevan · 2017-02-04T08:49:33Z

@mach6

Here's how the apis and their corresponding responses would look like

http://localhost:4444/api/v1

GET endpoints:

/api/v1
- Description: HTML page documenting the API endpoints (this page)
- Class: org.openqa.grid.web.servlet.api.v1.ApiV1
- Usage: /api/v1
/api/v1/hub
- Description: Returns configuration and proxy information of the hub
- Class: org.openqa.grid.web.servlet.api.v1.HubInfo
- Usage: /api/v1/hub
/api/v1/nodes
- Description: List of computers connected to the HUB and the nodes on each computer
- Class: org.openqa.grid.web.servlet.api.v1.Nodes
- Usage: /api/v1/nodes
/api/v1/proxy
- Description: Returns configuration and capability information for the current proxy
- Class: org.openqa.grid.web.servlet.api.v1.Proxy
- Usage: /api/v1/proxy/<ID> - ID retrieved from the /node endpoint
/api/v1/sessions
- Description: List of currently open sessions
- Class: org.openqa.grid.web.servlet.api.v1.Sessions
- Usage: /api/v1/sessions
/api/v1/session
- Description: Returns details for a given session
- Class: org.openqa.grid.web.servlet.api.v1.SessionInfo
- Usage: /api/v1/sessions/ - ID retrieved from the /sessions endpoint
Get Hub information. http://localhost:4444/api/v1/hub

{
  "configuration": {
    "capabilityMatcher": "org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
    "newSessionWaitTimeout": -1,
    "throwOnCapabilityNotPresent": true,
    "cleanUpCycle": 5000,
    "custom": {},
    "host": "192.168.1.5",
    "servlets": [],
    "withoutServlets": [],
    "browserTimeout": 0,
    "debug": false,
    "port": 4444,
    "role": "hub",
    "timeout": 1800
  },
  "nodes": [
    {
      "host": "192.168.1.5",
      "port": 5555,
      "id": "http://192.168.1.5:5555"
    },
    {
      "host": "192.168.1.5",
      "port": 6666,
      "id": "http://192.168.1.5:6666"
    }
  ],
  "registrationUrl": "http://192.168.1.5:4444//grid/register/",
  "consoleUrl": "http://192.168.1.5:4444//grid/console",
  "newSessionRequestCount": 0,
  "usedProxyCount": 2,
  "totalProxyCount": 2,
  "activeSessionCount": 3
}

Get list of nodes attached to the hub http://localhost:4444/api/v1/nodes

[
  {
    "host": "192.168.1.5",
    "port": 5555,
    "id": "http://192.168.1.5:5555",
    "slotUsage": {
      "total": 11,
      "busy": 2,
      "breakup": [
        {
          "browser": "chrome",
          "usage": {
            "total": 5,
            "busy": 2
          }
        },
        {
          "browser": "internet explorer",
          "usage": {
            "total": 1,
            "busy": 0
          }
        },
        {
          "browser": "firefox",
          "usage": {
            "total": 5,
            "busy": 0
          }
        }
      ]
    }
  },
  {
    "host": "192.168.1.5",
    "port": 6666,
    "id": "http://192.168.1.5:6666",
    "slotUsage": {
      "total": 11,
      "busy": 1,
      "breakup": [
        {
          "browser": "chrome",
          "usage": {
            "total": 5,
            "busy": 1
          }
        },
        {
          "browser": "internet explorer",
          "usage": {
            "total": 1,
            "busy": 0
          }
        },
        {
          "browser": "firefox",
          "usage": {
            "total": 5,
            "busy": 0
          }
        }
      ]
    }
  }
]

Given a Proxy Id, get details of the proxy http://localhost:4444/api/v1/proxy/http://192.168.1.5:5555

{
  "isBusy": true,
  "sessions": [
    {
      "browserName": "chrome",
      "sessionId": "157569ab-e0d8-4d04-b78d-7244d9d2d0c4"
    },
    {
      "browserName": "chrome",
      "sessionId": "30c089bd-ff76-4b95-a157-78519c488986"
    }
  ],
  "java": {
    "version": "1.8.0_66"
  },
  "os": {
    "name": "Mac OS X",
    "arch": "x86_64",
    "version": "10.12.3"
  },
  "slotUsage": {
    "total": 11,
    "busy": 2,
    "breakup": [
      {
        "browser": "chrome",
        "usage": {
          "total": 5,
          "busy": 2
        }
      },
      {
        "browser": "internet explorer",
        "usage": {
          "total": 1,
          "busy": 0
        }
      },
      {
        "browser": "firefox",
        "usage": {
          "total": 5,
          "busy": 0
        }
      }
    ]
  },
  "htmlRenderer": "org.openqa.grid.selenium.proxy.WebProxyHtmlRenderer",
  "build": {
    "version": "3.0.1",
    "revision": "unknown",
    "time": "unknown"
  },
  "percentUsed": 40.0,
  "lastSessionStart": 1486196956545,
  "config": {
    "hubHost": "localhost",
    "hubPort": 4444,
    "id": "http://192.168.1.5:5555",
    "capabilities": [
      {
        "browserName": "chrome",
        "seleniumProtocol": "WebDriver",
        "maxInstances": 5,
        "platform": "MAC"
      },
      {
        "browserName": "firefox",
        "seleniumProtocol": "WebDriver",
        "maxInstances": 5,
        "platform": "MAC"
      },
      {
        "browserName": "internet explorer",
        "seleniumProtocol": "WebDriver",
        "maxInstances": 1,
        "platform": "MAC"
      }
    ],
    "downPollingLimit": 2,
    "hub": "http://localhost:4444/grid/register",
    "nodePolling": 5000,
    "nodeStatusCheckTimeout": 5000,
    "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
    "register": true,
    "registerCycle": 5000,
    "unregisterIfStillDownAfter": 60000,
    "cleanUpCycle": 5000,
    "custom": {},
    "host": "192.168.1.5",
    "maxSession": 5,
    "servlets": [],
    "withoutServlets": [],
    "browserTimeout": 0,
    "debug": false,
    "port": 5555,
    "role": "node",
    "timeout": 1800
  }
}

List all active sessions in the Hub http://localhost:4444/api/v1/sessions

{
  "31006720-cddd-408d-8f65-09813b8b5cb7": {
    "slotInfo": {
      "host": "192.168.1.5",
      "port": 6666,
      "id": "http://192.168.1.5:6666"
    },
    "browser": "chrome"
  },
  "eb38d217-3445-4f03-bf58-3428d28fb130": {
    "slotInfo": {
      "host": "192.168.1.5",
      "port": 5555,
      "id": "http://192.168.1.5:5555"
    },
    "browser": "chrome"
  },
  "8e9b194d-72ce-4b69-aecf-a0672f854b57": {
    "slotInfo": {
      "host": "192.168.1.5",
      "port": 5555,
      "id": "http://192.168.1.5:5555"
    },
    "browser": "chrome"
  }
}

Given a Session Id, get details of the session http://localhost:4444/api/v1/session/8e9b194d-72ce-4b69-aecf-a0672f854b57

{
  "isForwardingRequest": false,
  "proxy": {
    "host": "192.168.1.5",
    "port": 5555,
    "nodeId": "http://192.168.1.5:5555"
  },
  "protocol": "WebDriver",
  "lastActivityWasAt": 253698,
  "isOrphaned": false,
  "internalKey": "bb556e03-95c4-4abd-bbae-7fbcfe16c026",
  "requestedCapabilities": {
    "browserName": "chrome"
  },
  "requestPath": "/wd/hub"
}

mach6 · 2017-02-06T19:24:35Z

@krmahadevan thanks for this! This is going to take a little time to vet with the Se community. I'll look at the existing work in more detail in the days to come but in the meantime, will you please add tests? Cheers!

krmahadevan · 2017-02-24T17:06:49Z

@mach6 - Any news on how is the review with the SE community going on ? Thought of quickly checking on this!

mach6 · 2017-02-24T23:28:49Z

java/server/src/org/openqa/grid/web/servlet/api/v1/HubInfo.java

Needs the copyright text

mach6 · 2017-02-24T23:34:46Z

@krmahadevan in general, agreement that since all of these api's are GET only, it is okay and will add value -- though there is concern about the long term implications of supporting additional api's and the fact that we'll have a mixed bag of RESTful (GET) operations and RESTish (POST/DELETE/GET) operations (the existing apis).

Once we complete the tests and remaining refinement, I believe I can merge this.

mach6 · 2017-02-24T23:37:21Z

java/server/src/org/openqa/grid/web/servlet/api/v1/utils/ProxyUtil.java

I liked it better when we were hashing the Id for the benefit of the HTTP URL. This looks silly to me http://localhost:4444/api/v1/proxy/http://192.168.1.5:5555 due to http:// ... http://

Fixed this by accepting both forms viz., with and without the protocol

krmahadevan · 2017-02-27T04:46:05Z

@mach6 - I have added tests as well. We should now be good to get this merged.

krmahadevan · 2017-02-28T04:14:57Z

ping @mach6 .. Gentle reminder Doug!

mach6 · 2017-02-28T06:40:04Z

java/server/test/org/openqa/grid/e2e/misc/GridRestfulAPITest.java

needs the copyright

mach6 · 2017-02-28T06:56:26Z

java/server/src/org/openqa/grid/web/servlet/api/v1/Proxy.java

The problem is a proxy id can be anything a user wants it to be. Only by default or if the (user followed the default convention) will it be http://{host}:{port}

If you look at java -jar selenium-server-standalone -role node -h and the documentation for id it says this;

-id <String> : optional unique identifier for the node. Defaults to the url of the remoteHost

Meaning it can be http://ftp://$$$123/*hello/@123/rdp:// or any crazy text I want it to be -- as long as it is unique. :)

So, we really need to one way hash the proxy id or similar and require the user to match all proxy id's to their hash by making a GET on a /proxies resource (or I guess it could be the /nodes resource)..

Come to think of it -- perhaps it is confusing having /nodes and /proxy/{id} .. we probably should combine these into one.

Something like;

/proxies -- get a list of all proxies
/proxies/{id} - get for the specified proxy

Similarly instead of /session and /sessions perhaps this would be better;

/sessions -- get a list of all sessions
/sessions/{id} - get for the specified session

@mach6

The problem is a proxy id can be anything a user wants it to be. Only by default or if the (user followed the default convention) will it be http://{host}:{port}

In that case the earlier change itself was sufficient, because it was basically querying based on whatever the user passes. So we can revert it back.

So, we really need to one way hash the proxy id or similar and require the user to match all proxy id's to their hash by making a GET on a /proxies resource (or I guess it could be the /nodes resource)..

This additional step IMHO Its not user friendly. I don't think there is anything wrong with using whatever the user has provided as the id for a proxy (even if it means that we are going to default to the url of the remoteHost) which should also be fine IMO rather than resorting to hashing it and making it more complex.

If you look at java -jar selenium-server-standalone -role node -h and the documentation for id it says this;

I don't remember seeing this in Selenium 2.53.1. Following is what I see

java -jar selenium-server-standalone-2.53.1.jar -role node -help ------------------------------- Running as a grid node: ------------------------------- Usage: java -jar selenium-server.jar -role node [options] -host: <IP | hostname> : usually not needed and determined automatically. For exotic network configuration, network with VPN, specifying the host might be necessary. -port: <xxxx> : the port the remote/hub will listen on. Default to 4444. -cleanupCycle: <XXXX> in ms. How often a proxy will check for timed out thread. -timeout: <XXXX> the timeout in seconds before the hub automatically ends a test that hasn't had any activity in the last X seconds. The browser will be released for another test to use. This typically takes care of the client crashes. -browserTimeout: The timeout in seconds a browser can hang -hub: <http://localhost:4444/grid/register> : the url that will be used to post the registration request. This option takes precedence over -hubHost and -hubPort options. -hubHost: <IP | hostname> : the host address of a hub the registration request should be sent to. Default to localhost. Option -hub takes precedence over this option. -hubPort: <xxxx> : the port listened by a hub the registration request should be sent to. Default to 4444. Option -hub takes precedence over this option. -proxy: the class that will be used to represent the node. By default org.openqa.grid.selenium.proxy.DefaultRemoteProxy. -maxSession: max number of tests that can run at the same time on the node, independently of the browser used. -registerCycle: how often in ms the node will try to register itself again.Allow to restart the hub without having to restart the nodes. -nodePolling: in ms. Interval between alive checks of node how often the hub checks if the node is still alive. -unregisterIfStillDownAfter: in ms. If the node remains down for more than unregisterIfStillDownAfter millisec, it will disappear from the hub.Default is 1min. -downPollingLimit: node is marked as down after downPollingLimit alive checks. -nodeStatusCheckTimeout: in ms. Connection and socket timeout which is used for node alive check. This synopsis lists options available in node role only. To get help on the command line options available for other roles run the server with -help name and the corresponding -role name value.

Just thinking aloud here. I am not sure if this is something that should be left to the user to customize because this is internally used by the node to uniquely identify itself (which you called out). In that case, should a user really fiddle around with this ? Its not altering or tweaking the behavior of the node in any way (which I believe is what the different parameters in the GridNodeConfiguration stand for. So all that said and done, is this something that we should really worry about ? Thoughts ?

In that case the earlier change itself was sufficient, because it was basically querying based on whatever the user passes. So we can revert it back.

Well, that makes the HTTP GET URL look silly again. I'll think about / bounce the consideration off of others a little more ..

Just thinking aloud here. I am not sure if this is something that should be left to the user to customize because this is internally used by the node to uniquely identify itself (which you called out). In that case, should a user really fiddle around with this

The hub registration protocol makes these allotments. There are use cases where 'autodetecting' or 'assuming' the proxy id is not sufficient. In Se2.x the allotments were also there -- except you could only specify proxy id via code or the config json, if I recall correctly.

There are use cases where 'autodetecting' or 'assuming' the proxy id is not sufficient.

@mach6 - Can you please help call out what those use cases might be ? I would be curious to learn/hear about them

mach6 · 2017-02-28T07:00:33Z

@krmahadevan thx for the follow-up here. I left a few more new comments for your consideration

Following was done : * Cleaned up the end-points. * Added a new end-point for querying hub information.

Consolidated implementations for the end-points for proxies and sessions and updated tests accordingly.

krmahadevan · 2017-03-04T15:44:55Z

@mach6 - I have consolidated the proxy end-point and the session end-point into the same thing so that it works for querying all proxies/sessions (or) for a particular proxy/session.

I would really like to get this PR wrapped up. Please let me know what else is pending on this.

krmahadevan · 2017-03-09T03:21:48Z

ping @mach6 - Gentle reminder!

krmahadevan · 2017-03-13T17:53:50Z

ping @mach6 - Any updates on this ?

krmahadevan · 2017-03-14T15:46:29Z

@lukeis - Do you think you could help me with this ? I haven't heard from @mach6 Not sure why though! Its been more than a month since this PR was raised.. and I would really love to have this feature within Grid...

krmahadevan · 2017-05-18T06:42:46Z

@mach6 - Pls let me know if I should just close off this PR. I haven't heard of anything on this PR and its more than a couple of months...

mach6 · 2017-05-19T04:19:47Z

Sorry for the continued delay. please do not close

krmahadevan · 2017-05-26T14:36:19Z

@mach6 - What is holding this PR ? Can you please let me know. I was hoping that this would be wrapped up a long time back.

Following was done : * Cleaned up the end-points. * Added a new end-point for querying hub information. * Consolidated implementations for the end-points for proxies and sessions and updated tests accordingly. Signed-off-by: Doug Simmons <doug@mach6.net>

krmahadevan mentioned this pull request Feb 4, 2017

Enhance the "grid/api/hub" response to include the list of nodes wired into the Hub as well #3446

Closed

krmahadevan force-pushed the krmahadevan_github_265 branch from 8e2fa51 to aa151e9 Compare February 4, 2017 08:56

mach6 added the B-grid Everything grid and server related label Feb 6, 2017

mach6 reviewed Feb 24, 2017

View reviewed changes

krmahadevan force-pushed the krmahadevan_github_265 branch 2 times, most recently from 03623c1 to b50f820 Compare February 27, 2017 04:51

mach6 reviewed Feb 28, 2017

View reviewed changes

java/server/test/org/openqa/grid/e2e/misc/GridRestfulAPITest.java Outdated

Copy link
Copy Markdown

Member

mach6 Feb 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs the copyright

mach6 reviewed Feb 28, 2017

View reviewed changes

dima-groupon and others added 5 commits March 4, 2017 21:07

Adding POC for REST api on Selenium Hub

9600815

[CHECKPOINT] Revive SeleniumHQ#265 for Se3

45ec866

[CHECKPOINT] no need for apache-commons

3e2c4c4

Wrapping up the REST API Changes

04f6325

Following was done : * Cleaned up the end-points. * Added a new end-point for querying hub information.

Fixing review comments and added tests.

7c55098

krmahadevan force-pushed the krmahadevan_github_265 branch from b50f820 to 09e74b2 Compare March 4, 2017 15:39

Fixing additional review comments.

961c181

Consolidated implementations for the end-points for proxies and sessions and updated tests accordingly.

krmahadevan force-pushed the krmahadevan_github_265 branch from 09e74b2 to 961c181 Compare March 4, 2017 15:41

krmahadevan closed this Jun 8, 2017

krmahadevan deleted the krmahadevan_github_265 branch June 8, 2017 04:33

Uh oh!

Conversation

krmahadevan commented Feb 4, 2017

Uh oh!

krmahadevan commented Feb 4, 2017

Uh oh!

krmahadevan commented Feb 4, 2017

Uh oh!

mach6 commented Feb 6, 2017

Uh oh!

krmahadevan commented Feb 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mach6 commented Feb 24, 2017

Uh oh!

mach6 Feb 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krmahadevan commented Feb 27, 2017

Uh oh!

krmahadevan commented Feb 28, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mach6 commented Feb 28, 2017

Uh oh!

krmahadevan commented Mar 4, 2017

Uh oh!

krmahadevan commented Mar 9, 2017

Uh oh!

krmahadevan commented Mar 13, 2017

Uh oh!

krmahadevan commented Mar 14, 2017

Uh oh!

krmahadevan commented May 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mach6 commented May 19, 2017

Uh oh!

krmahadevan commented May 26, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mach6 Feb 24, 2017 •

edited

Loading

krmahadevan commented May 18, 2017 •

edited

Loading