NSE for OpenWebNet discovery #915

rewanthtammana · 2017-06-21T15:36:15Z

The following script discovers the services running on OpenWebNet protocol.

The script currently fetches the Gateway address, Device type, Number of devices running, Addresses of all services. Handles the errors like Socket connection error, EOF as response and timeout errors.

I have tested it on 4 Indian servers and handled the errors properly.

Do let me know if anything needs to be changed.

dmiller-nmap

A good start. Here are a few things to consider.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+-- |   Burglar Alarm: 1
+-- |_  Lighting: 114
+--
+-- Version: 0.1, Updated on 21/06/2017


Don't do internal versioning. Git/SVN handles this for us, and it just ends up being clutter in the file.

Patched as 540b75e

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+portrule = shortport.port_or_service(20000, "openwebnet")
+
+local device = {}
+device[2] = "MHServer"


The preferred syntax for this is to allocate and create the table all in one statement. This is slightly more efficient since Lua can know the size of the table right away and doesn't have to keep growing and reallocating as we add elements:

local device = { [2] = "MHServer", [4] = "MH200", -- etc. }

Patched as 540b75e

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+    return nil, nil, "Received a negative ACK as response."
+  end
+
+  stdnse.sleep(2)


What is the purpose of this sleep? Is it really needed, or are we just not synchronizing TCP streams very well?

I didn't get the responses properly when I made large number of requests. That's due to my poor internet connection which I realized now. So, I removed stdnse.sleep(2) from the code and its committed as 5fb09bb.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+  sd:send(request)
+
+  local status, data = sd:receive_buf("*#*1##", false)
+  if data == "EOF" then


Should be testing status here, not data. It doesn't matter what the error is, really, only whether we received a response or not. Pass the error string directly up the stack if you wish. Same below with "TIMEOUT"

Patched as c7e24f4.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+  end
+end
+
+-- Removes *#*1## from the beginning and ending


Rather than just trimming off the ACK responses, can we convert this to a readable string? If it's an IP address, extracting the data (after the WHO number) and replacing "#" with "." seems to be the way to do it. Other data types will require other parsing.

It confirms that the data is received completely for a particular request.

What do you mean by if it's an IP address? , the response pattern will be same in all the cases, isn't it?

Let's take the request for the device's own IP address as an example, since the reference page says that Gateway IP is only sometimes supported. The request is *#13**10## and the response is:

*#13**10*192*168*1*40##*#*1## ||| || | | |_ ACK ||| || | |_ End of message ||| || |_ VALUE: 192.168.1.40 ||| ||_ DIMENSION: IP Address ||| |_ WHERE: empty |||_ WHO: Device Communication ||_ this is a Dimension message |_ Start of message

So for any request, we should parse the response into a series of messages (usually there will be only 1 plus an ACK), validate that it corresponds to the request we sent (WHO, WHERE, and DIMENSION match), and then convert the VALUE as appropriate. In this case, it's clear that the value is a list of the octets of an IP address. We could probably just replace the * with . in that case. Time and Date are also interesting, so we'll have to extract each value and convert it into a readable string. When we find an ACK or NACK message, that means we're done parsing; NACK means the command was bad or not supported.

Committed as ff5d5ec.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+
+  local output = {}
+
+  local sd, gateway, err = get_socket(host, port, "*#13**15##")


Instead of directly passing OpenWebNet command strings (with "*#" and all that), write a function that returns the appropriate command like so:

local command = get_command("Device Communication", "Gateway IP Address")

This would do a lookup in a table to convert the first parameter to "13" and the second to "50". Note that what you are requesting here is actually the device type (15), not the gateway (50) as the script indicates.

But how can i know whether the parameters in the command are referring to WHO or WHAT or WHERE? How should I differentiate that?

In this case, I think its better to use something like this,
local command = get_command(WHO, WHERE, WHAT)

Your opinion please.

I see that could be difficult. Then I would have separate functions for each of the message types we want to support. I think this will only be Status Request (WHO, WHERE) and Dimension Request (WHO, WHERE, DIMENSION). In fact, we can support both with a single function which simply joins all arguments (...) with * and adds the message start (*# for requests) and end (##).

I managed to code it this way, 2ed6404. Your thoughts on this implementation please?

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+
+-- Returns table after appending the delimiter
+-- The return table contains the list of devices
+local function custom_split(delimiter, resultant)


Instead of this function, you can call socket:receive_buf(delimiter, false) and it will return the next chunk of data up to the delimiter from the socket's receive buffer. In a loop, you can watch for an ACK message to indicate the end of the list, so that you stop trying to receive.

I tried doing this, but there are many duplicate results if I used ## as a delimiter. I think keeping this separate function will do a better job as we can get the data from the server in one shot and we can perform small operations on it to extract the results.

I think that the duplicate results are caused by not taking into account the ACK messages which frame the response: there's one at the beginning as a banner, then one at the end of each response (or a NACK if the request couldn't be answered). The syntax does not otherwise allow ## anywhere else in a message.

Its working now. Committed as 6e7276c.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+  end
+
+  -- Fetching list of each device
+  for _, v in pairs(who) do


I don't think we should do this for every possible WHO value. Some of them may have other functions than "get the status of all X" if you call them with a WHAT of 0. For example, Device Communication (13) responds with a NACK. Instead, keep a list of ones that do respond properly like this (1 through 25, maybe, although I don't know what CEN means) and check those this way.

Committed as ceb0968.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+
+    stdnse.debug("Fetching the list of " .. v .. " devices.")
+
+    local sd, data, err = get_socket(host, port, "*##*#" .. _ .. "*0##")


What happened to the last socket we opened? If we keep the TCP stream synchronized (meaning we know which responses belong to which requests), we should be able to keep sending requests/commands on the same socket.

If I'm using the last socket we opened, I'm getting the previous response only even after sending the new data. That's the reason I'm creating new socket every time. I'm not sure of the reason behind this kind of response.

Try it again after implementing the parsing I suggested. socket:receive_buf ought to make it simple to only consume the buffer once; as long as you only send one request and then read up to the first ACK or NACK, the socket should be ready for a second request and response.

Committed as 6e7276c.

dmiller-nmap · 2017-06-21T20:37:28Z

scripts/openwebnet-discovery.nse

+
+action = function(host, port)
+
+  local output = {}


Make output a stdnse.output_table(), so that the keys are always in the same order. Some software like Ndiff will treat different orderings as different results.

Patched as 4d0d4a9.

dmiller-nmap

Great! These are mostly formatting changes; the protocol implementation looks fine.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+  [4] = "Heating",
+  [5] = "Burglar Alarm",
+  [6] = "Door Entry System"
+}


It'd be nice to still have the full list of WHO values, even if we only query 0 through 6.

Committed as aedb765.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+}
+
+local device_dimensions = {
+  ["Time"] = "*#13**0##",


It would be cleaner to have this table simply be the dimension numbers, and use a function to add the "*#13**" and "##"

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+OpenWebNet is a communications protocol developed by Bticino since 2000.
+Retrieves the Gateway and device type. Retrieves the count and addresses
+of lights, multimedia and many other services running on server/servers.
+]]


Add some References here: the piMyHome link and https://www.myopen-legrandgroup.com/solution-gallery/openwebnet/

Committed as 337eda5.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+
+    -- If response is NACK, it means the request method is not supported
+    if data == NACK then
+      res = {}


Returning nil is easier to check for than empty table. It's OK if WHO categories that have 0 devices are not shown in the script output.

I think showing 0 devices will be more helpful to the user during reconnaissance.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+
+local function format_dimensions(res)
+
+  if res["Time"] then


For "Time" and "Date", it would be preferable to send the "Date and Time" (22) query, since it saves a query. Then extract the year, month, day, hour, minute, second and pass them in a table to stdnse.format_timestamp. The table format is described in the Lua manual for os.time.

Committed as 796e6dc.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+
+    stdnse.debug("Fetching the list of " .. v .. " devices.")
+
+    local res = get_response(sd, "*##*#" .. _ .. "*0##")


Don't send the "*##" empty/invalid command first.

Committed as 39e9980.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+  end
+
+  -- Fetching list of dimensions of a device
+  for _, v in pairs(device_dimensions) do


pairs does not produce a predictable ordering, but we need output to be predictable. I suggest completing the device_dimensions table to include all values, even ones we do not query. Then iterate over a separate table of dimension names that we do want.

Only use _ as a loop variable when you intend to not use it elsewhere. It's the "throwaway" variable. If you intend on using it, use a descriptive name. Here's an example of what I am talking about. _ is used for the index, which we don't care about; we only want the names in order.

for _, label in ipairs({"Device Type", "Date and Time", "Uptime", "Firmware version", ...}) do

Committed as 5de72e6.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+
+description = [[
+OpenWebNet is a communications protocol developed by Bticino since 2000.
+Retrieves the Gateway and device type. Retrieves the count and addresses


We don't retrieve the Gateway or "addresses of lights". Generalize this description to only say we retrieve device identifying information and number of connected devices.

Committed as 337eda5.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+  end
+
+  if res["MAC address"] then
+    res["MAC address"] = string.gsub(res["MAC address"], "%.", "-")


We want the MAC address as colon-separated hex. Here's a list of functions that may help:

stdnse.tohex

stdnse.format_mac

string.format("%x")

table.concat

string.gsub (with function instead of replacement string)

Committed as f563a23.

dmiller-nmap · 2017-06-27T17:17:13Z

scripts/openwebnet-discovery.nse

+--  |   Device Type: F453AV
+--  |   Distribution Version: 3.0.1
+--  |   Firmware version: 3.0.14
+--  |   Uptime: 5.3.28.38


Uptime should be reported in the same format that stdnse.format_difftime uses. You don't have to use that function, though, since it's already reported broken into segments like this. Your example output should look like "5d3h28m38s". Note that the piMyHome page does not show seconds, so some devices may not report that.

Committed as be53037.

…ons.

rewanthtammana · 2017-07-01T21:32:43Z

@dmiller-nmap, All the requested changes are made, your final views on the modified code ?

dmiller-nmap

This feedback is based on running the script; it still takes almost 20 seconds to run after I made some hasty fixes regarding NACK handling, but it's a big improvement over the current version.

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+Retrieves device identifying information and number of connected devices.
+
+References:
+  https://www.myopen-legrandgroup.com/solution-gallery/openwebnet/


Use * at the beginning of NSEdoc lines to indicate an unordered list.

Committed as 25456ad

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+--  | openwebnet-discover:
+--  |   IP Address: 192.168.200.35
+--  |   Net Mask: 255.255.255.0
+--  |   MAC Address: 0:3:50:1:d3:11


This should be formatted as 00:03:50:01:d3:11. Hints on how to do this below.

Committed as 348e234

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+--  |   Distribution Version: 3.0.1
+--  |   Date: 02.07.2017
+--  |   Time: 02:11:58
+--  |   Scenarios: 0


I still prefer to not see lines with 0 count.

Committed as 0b5cde6

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+-- Returns the socket and error message
+local function get_socket(host, port, request)
+
+  local sd, response, early_resp = comm.opencon(host, port, request)


Since we expect to get a banner in response to connection, we need to set the recv_before option to opencon. This will connect, wait for a banner, and return that as early_resp. We should NOT be sending an ACK here, since it only responds with a NACK because ACK is not a client message. So the intended flow is:

C > S - connect

C < S - ACK

C > S - request

C < S - response

Committed as c573a9f

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+  local sd, response, early_resp = comm.opencon(host, port, request)
+
+  if sd == nil then
+    return nil, "Socket connection error."


Return or print (debug) the actual error, which is in the response variable now.

Committed as b1a2a9a

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+      table.insert(t, stdnse.tohex(tonumber(v)))
+    end
+
+    res["MAC Address"] = table.concat(t, ":")


Instead of splitting, converting, and joining, try using string.gsub with a function like so:

formatted = string.gsub(raw, "(%d+)%.?", function (number) return do_something(number) end )

You have options for do_something: you could simply return the byte with the value of number, which would convert the whole string to a 6-byte blob suitable for passing to stdnse.format_mac. You could convert it directly to hex with string.format, and change the separator to ":" either there or in a separate pass (not consuming it in the pattern). Etc.

Committed as 348e234

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+
+    for _, v in ipairs(stdnse.strsplit("%.%s*", res["Uptime"])) do
+      table.insert(t, v .. units[counter])
+      counter = counter + 1


Here, the _ variable is an index you could use instead of keeping a separate counter. Remember that Lua tables are 1-indexed, so remove the [0] = from the declaration of units at the top. Otherwise, this is a smart solution.

Committed as fa69e5a

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+
+    stdnse.debug("Fetching " .. device)
+
+    local res = get_response(sd, head .. device_dimension[device] .. tail)


Need to check that res is valid here. A good test for a table with values in it is: if res and next(res) then

Committed as 377da5d

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+    local regex = string.gsub(head, "*", "%%*") .. device_dimension[device] .. "%*" .."(.+)" .. tail
+    local tempRes = string.match(res[1], regex)
+
+    output[device] = string.gsub(tempRes, "*", ".")


Did we check whether tempRes is not nil? What if the server sends a response that doesn't match?

Committed as 377da5d

dmiller-nmap · 2017-07-03T17:41:37Z

scripts/openwebnet-discovery.nse

+  output = format_dimensions(output)
+
+  -- Fetching list of each device
+  for i = 0, 6 do


I don't think Scenarios (WHO=0) supports retrieving a status list like this. Just run from 1 to 6.

Committed as 50af9ca

rewanthtammana added 3 commits June 21, 2017 20:58

NSE script for OpenWebNet discovery

6aa0984

Ignored the timeout errored output for better view of result

2a4f8cb

Fixes Indentation

28d70ba

dmiller-nmap requested changes Jun 21, 2017

View changes

rewanthtammana added 12 commits June 22, 2017 20:48

Modifies tables initiation

540b75e

Checks status variable instead of data for status

c7e24f4

Uses stdnse.output_table()

4d0d4a9

Removes stdnse.sleep

5fb09bb

Uses only single socket

6e7276c

Reduces WHO test values

ceb0968

Removes match module

4865a6d

Sends Gateway request into generalized format

27ab35c

Adds generalized function for getting sending requests

669807d

Fetches the list of device dimensions

2ed6404

Formats the output of device dimensions

ff5d5ec

Removes unused functions

be446a2

dmiller-nmap requested changes Jun 27, 2017

View changes

rewanthtammana added 8 commits July 1, 2017 23:39

Adds references.

337eda5

Adds more WHO values.

aedb765

Corrects request message.

39e9980

Removes convoluted match and uses ipairs for iterating device dimensi…

5de72e6

…ons.

Uses single query for Date and Time.

796e6dc

Formats MAC Address.

f563a23

Formats Uptime.

be53037

Modifies @output.

79386c6

rewanthtammana added 2 commits July 2, 2017 03:13

Follows better naming conventions.

ee67c24

Minor fixes and updates done.

d6ee4ce

rewanthtammana force-pushed the openwebnet branch from 942347a to d6ee4ce Compare July 1, 2017 21:55

Updates device dimensions list.

7b123c2

dmiller-nmap requested changes Jul 3, 2017

View changes

rewanthtammana added 10 commits July 4, 2017 12:05

Adds * for references

25456ad

Formats MAC Address

348e234

Optimizes Uptime formatting

fa69e5a

Checks for valid response

377da5d

Ignores scenarios

50af9ca

Removes unused operations

127ea7c

Formats Date and Time


		local output = {}

		local sd, gateway, err = get_socket(host, port, "#13*15##")


		stdnse.debug("Fetching the list of " .. v .. " devices.")

		local sd, data, err = get_socket(host, port, "###" .. _ .. "*0##")


		stdnse.debug("Fetching the list of " .. v .. " devices.")

		local res = get_response(sd, "###" .. _ .. "*0##")


		stdnse.debug("Fetching " .. device)

		local res = get_response(sd, head .. device_dimension[device] .. tail)

NSE for OpenWebNet discovery #915

NSE for OpenWebNet discovery #915

Conversation

rewanthtammana commented Jun 21, 2017

dmiller-nmap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rewanthtammana Jun 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmiller-nmap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rewanthtammana Jul 1, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rewanthtammana commented Jul 1, 2017

dmiller-nmap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rewanthtammana Jun 24, 2017 •

edited

rewanthtammana Jul 1, 2017 •

edited