History support server side

[plieningerweb]

Hey,

first of all, awesome work guys! I am really happy this project exists 👍

In the last days I was using the opc ua server for some monitoring of energy usage.
Now, I got to the point that I can not report any errors which occurred while reading the energy stats in the opc ua server. As I understand it, reporting events like errors right now in OPC UA only work using the subscription or history method.

As I do only want to connect once a day to the opc ua server, i discarded the subscription method. (band with / data usage reasons)

What do you think?
Is there any plan or idea how to implement the history function for the opc ua server?

If I would get a better understanding and some help, maybe I could also start working on it.

[oroulet]

No plan yet, but you can give it a try!

I have been thinking a bit how to implement this. Here is a proposition

First you need to read the part of the spec explaining how histpry should behave. not so complicated I guess.

then you need to implement the method read_history in server, take example to one of the other methods like read. But in Session class you should check that history attribute is set and then call read_history on a History class that is instanciated as a member of InternalServer and that you implement in its own history.py file.

InternalServer class should have a set_history_manager() method allowing to pass custom History class implementing the UaHistory interface but with different backend. I propose to implement a default backend using sqlite.

finally add a method to Node class. for example enable_history() taking at a max time or/and max number of datachange to remember. This method should set the history attribute, subscribe to the node and save data to the History class.

Is it understandable?

[oroulet]

Are you looking at it?

[plieningerweb]

Right now I do not have enough time :( maybe next month.

[zerox1212]

I'm starting to add a basic SQLite history.

Can you explain what you are doing in this?

    def save_node_value(self, node, datavalue):
        data = self._datachanges[node]
        period = self._datachanges_period[node]
        data.append(datavalue)
        now = datetime.now()
        while now - data[0].ServerTimestamp > period:
            data.pop(0)

It looks like this method is where the data should be recorded when the subscriptions reports a data_change. However it seems this code just pops everything in the dictionary within the period.

[oroulet]

Great!
I am not finished so things are broken!
The idea is to have to methods for datachanges and to for events
new_node (very bad name, feel fri to propose something else) when you can
create a new table and save the 'period' which is the max amount of time to
save or
'count' which is the max number of changes to remember
then save_datachange which is called every time the node value changes.
after saving changes I check if last events is older than period of if the
number of changes is > count and remove the last entry if necessary

On Mon, 28 Mar 2016 at 18:31 Andrew notifications@github.com wrote:

I'm starting to add a basic SQLite history.

Can you explain what you are doing in this?

def save_node_value(self, node, datavalue):
    data = self._datachanges[node]
    period = self._datachanges_period[node]
    data.append(datavalue)
    now = datetime.now()
    while now - data[0].ServerTimestamp > period:
        data.pop(0)```

It looks like this method is where the data should be recorded when the subscriptions reports a data_change. However it looks like this code just pops everything in the dictionary within the period.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

I spent a bit of time on this today. It isn't functional yet but you can take look here zerox1212@47fb0de if you are interested. I will try to get it working tomorrow.

I found a nice tool called DB Browser for SQLite to view the data in the DB easier as well.

[zerox1212]

It turns out that sqlite's connection object has to be used on the same thread in which it's created.

Are all subscription call backs always on the same thread? That would make it a lot easier because all data_change events can use the same sqlite connection object to insert data into the database.

[zerox1212]

I have implemented basic recording part of the history on my fork more or less. I tested it some as well.
https://github.com/zerox1212/python-opcua/commits/history

However when I started to build the read history stuff and test it I ran into problems.

1. I don't know what read_datavalue_history is supposed to return. Is it a list of UA variants?
2. I will assume your "history" branch doesn't have a way for clients to call history_manager.read_history yet. Is this true?
3. My test client shows that the node historizing value as TRUE, but if I try to add the node to the client historian it just tells me that this node doesn't support historizing. Most likely because of point 2.

I still have a hard time navigating the core of the OPC UA library, do you have an idea when you can finish implementing the OPC UA part? That way I can finish the SQLite portion.

Thanks

[oroulet]

Hi,
I implemented a bit more of history support and added some tests. You can
now test and try you sqlite backend with it. I also did some documentation
and renaming.

On Tue, 29 Mar 2016 at 05:19 Andrew notifications@github.com wrote:

I have implemented the recording part of the history more or less on my
fork.

However when I started to build the read history stuff and test it I ran
into problems.

1. I don't know what read_datavalue_history is supposed to return. Is
   it a list of UA variants?
2. I will assume your "history" branch doesn't have a way for clients
   to call history_manager.read_history yet. Is this true?
3. My test client shows that the node historizing value as TRUE, but
   if I try to add the node to the client historian it just tells me that this
   node doesn't support historizing. Most likely because of point 2.

I still have a hard time navigating the core of the OPC UA library, do you
have an idea when you can finish implementing the OPC UA part? That way I
can finish the SQLite portion.

Thanks

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[oroulet]

look at the tests to see how to use the api

On Wed, 30 Mar 2016 at 21:12 Olivier Roulet-Dubonnet <
olivier.roulet@gmail.com> wrote:

Hi,
I implemented a bit more of history support and added some tests. You can
now test and try you sqlite backend with it. I also did some documentation
and renaming.

On Tue, 29 Mar 2016 at 05:19 Andrew notifications@github.com wrote:

I have implemented the recording part of the history more or less on my
fork.

However when I started to build the read history stuff and test it I ran
into problems.

1. I don't know what read_datavalue_history is supposed to return. Is
   it a list of UA variants?
2. I will assume your "history" branch doesn't have a way for clients
   to call history_manager.read_history yet. Is this true?
3. My test client shows that the node historizing value as TRUE, but
   if I try to add the node to the client historian it just tells me that this
   node doesn't support historizing. Most likely because of point 2.

I still have a hard time navigating the core of the OPC UA library, do
you have an idea when you can finish implementing the OPC UA part? That way
I can finish the SQLite portion.

Thanks

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

I will check it tonight and try to make a first implementation of SQLite.

Thanks.

[zerox1212]

I put the SQL implementation on hold for now because I want to make clients be able to access your minimal history.

When I check Part 11 of the spec it says that the node requires:

HasComponent: AggregateConfiguration (AggregateConifgurationType)
HasProperty: Stepped (Bool)

When i look at node.py, it is only possible to get_properties, not add. How can I extend our node class to have these items?

[oroulet]

The client is already able to read history. Look at uahistory tool and
test_history tests. The client can also read history from other servers.

On Fri, Apr 1, 2016, 17:45 Andrew notifications@github.com wrote:

I put the SQL implementation on hold for now because I want to make
clients be able to access your minimal history.

When I check Part 11 of the spec it says that the node requires:

HasComponent: AggregateConfiguration
HasProperty: Stepped

When i look at node.py, it is only possible to get_properties, not add.
How can I extend our node class to have these items?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

I'm trying to read my OPC UA server's history from UaExpert test client.

I get this message:

Get stepped property of node 2002 failed with status code BadNoMatch.

In my test code I added this property to my Variable. I will keep trying.

[zerox1212]

I partially solved the issue.

UaExpert checks the node UserAccessLevel for "ReadHistory". We should make sure the python client also respects these access levels for history.

Now I just need to figure out why UaExpert makes a datetime overflow when trying to read the history.

console from when UaExpert tries to do HistoryReadRawModified:

datetime overflow: -8344197343851053056
Cleanup client connection: ('127.0.0.1', 50072)
Exception raised while parsing message from client, closing
Traceback (most recent call last):
File "..\opcua\server\binary_server_asyncio.py", line 85, in _process_data
ret = self.processor.process(hdr, buf)
...................................
File "..\opcua\ua\uatypes.py", line 230, in unpack_bytes
return data.read(length)
File "..\opcua\common\utils.py", line 61, in read
raise NotEnoughData("Not enough data left in buffer, request for {}, we have {}".format(size, self))
opcua.common.utils.NotEnoughData: Not enough data left in buffer, request for 655360465, we have Buffer(size:30, data:b'\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x02\xd2\x07\xff\xff\xff\xff\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff')

[oroulet]

Ok thanks, I haven't tested that the server works with other clients..

On Fri, Apr 1, 2016, 18:24 Andrew notifications@github.com wrote:

I partially solved the issue.

UaExpert checks the node UserAccessLevel for "ReadHistory". We should make
sure the python client also respects these access levels for history.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[oroulet]

just pushed a commit that make server history works with uaexpert. It was a
long standing bug in default access level

On Fri, 1 Apr 2016 at 18:34 Olivier Roulet-Dubonnet <
olivier.roulet@gmail.com> wrote:

Ok thanks, I haven't tested that the server works with other clients..

On Fri, Apr 1, 2016, 18:24 Andrew notifications@github.com wrote:

I partially solved the issue.

UaExpert checks the node UserAccessLevel for "ReadHistory". We should
make sure the python client also respects these access levels for history.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#129 (comment)

[oroulet]

I now merged a working version, try to make you history sql to work with it

[oroulet]

@zerox1212 you may get issue getting your branch updated against master, sorry for the trouble, but now history is merged on master and you can safely work against it, instead of a moving branch

[zerox1212]

@oroulet No problem. I don't know git that well and so far I have not found out how to resynchronize my fork with the main branch. I end up deleting my fork, then forking again so it's up to date again. :(

I'll get back on SQL now thanks for the bug fixes.

[zerox1212]

Is there any reason for only passing the NodeId to the history interface?

    def historize(self, node, period=timedelta(days=7), count=0):
        ...
        self.storage.new_historized_node(node.NodeId, period, count)
        ...

I think we should always pass the entire node to the interface in case the history interface wants to gather more attributes about the node. Passing only node.NodeId makes it difficult to get the required information when storing in SQL. For example in my implementation of new_historized_node I can't do node.get_browse_name(), etc., for table or column names.

[oroulet]

We can pass a node object if we have a good reason for it. But the only
role of storage interfaces is to store what is necessary to reply to
history read request, nothing more. Why would you need the browser name of
a node?

On Fri, Apr 8, 2016, 01:22 Andrew notifications@github.com wrote:

Is there any reason for only passing the nodeid to the history interface?

def historize(self, node, period=timedelta(days=7), count=0):
    if not self._sub:
        self._sub = self._create_subscription(SubHandler(self.storage))
    if node in self._handlers:
        raise ua.UaError("Node {} is allready historized".format(node))
    self.storage.new_historized_node(node.nodeid, period, count)
    handler = self._sub.subscribe_data_change(node)
    self._handlers[node] = handler

I think we should always pass the entire node to the interface in case the
history interface wants to gather more attributes about the node.
self.storage.new_historized_node(node.nodeid, period, count)
Makes it difficult to get the required information when storing in SQL.
For example in my implementation of new_historized_node I can't do
node.get_browse_name(), etc.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

I understand. When I wrote that I was thinking of making a more human readable history database in case other software wanted to look at the data. Now I realize that this database should only ever be accessed via OPC UA methods.

Edit: After looking at it I still need the node itself in my interface.

The reason is that I must be able to do node.get_data_type() so that I can synchronize the SQL database type with the node type. If the history interface only gets the NodeId SQL won't work.

[zerox1212]

Made a pull request here: #155

I only tested it with one type of variable, so maybe someone else can run tests. (I don't really know how to run the automated tests yet)

Posted a few questions in the comments.

Also, can someone take a look here: https://docs.python.org/3.4/library/sqlite3.html and tell me the best way to store different OPC UA variants in SQLite3?

For now I'm converting the variant to an SQL type, but this will definitely result in a loss of precision.

[zerox1212]

Now that we have some history stuff going, I have a question.

Is it possible that when server.start() is called that we can loop through the address space and automatically call enable_history on any nodes that are set to historize?

I think it might be a bit easier to manage historizing this way. This would also mean we can add the "historizing" attribute to XML import to historize nodes without calling enable_history explicitly.

[oroulet]

we can loop through the address space and automatically call enable_history on any nodes that are set to historize?

that sounds very heavy...but maybe we should allow it so people can use xml to specify historization....Anyway the first step is to allow to call start() after enable_history(), it is currently not possible....and then we nee to implement events but we should merge #133 first

[destogl]

IMHO: I find the idea to automatically go through nodes and enable history on them, but hisotorizing should be somehow enabled, if not calling (global )enable_history at least have server parameter...

[oroulet]

IMHO: I find the idea to automatically go through nodes and enable history on them, but hisotorizing should be somehow enabled, if not calling (global )enable_history at least have server parameter...

history does not cost anything (ok a bit of memory) as long as it is not enabled for any nodes. but sqllite database need to be enabled if wanted

[zerox1212]

OK, no problem. I just wanted to bring it up as I have no idea how to implement it.

For now I can start taking a look at events instead.

[zerox1212]

So I made some progress on historizing events tonight, but I ran into a problem I want some input on.

If I set my node "Event Notifier" bit 2 to True to allow event HistoryRead, then try to subscribe to the event, the below code causes trouble because Value of the BYTE is being checked. The bit should be checked instead.

Code from internal_subscription.py line 94:

            if params.ItemToMonitor.AttributeId == ua.AttributeIds.EventNotifier:
                self.logger.info("request to subscribe to events for node %s and attribute %s", params.ItemToMonitor.NodeId, params.ItemToMonitor.AttributeId)
                if self.aspace.get_attribute_value(params.ItemToMonitor.NodeId, ua.AttributeIds.EventNotifier).Value.Value != 1:

This causes a bad status code to be returned even though SuscribeToEvents bit is set for this node.

[destogl]

Hi,

@zerox1212:

I'm now looking at storing events and I have a few questions. One thing I noticed with the latest code is that triggering the event multiple times doesn't update the time. Seems like the time is from the very first trigger() only. Is this correct?

This is known and I tried to solve it in #133 . I will try to finish this implementation today, than most of the errors should be fixed.

Actually, all of mentioned errors are known... Hopefully soon you have fixed version...

[zerox1212]

I don't fully understand your changes, but I have the first half of events history working (saves events, can't read them yet).

I made the interface so that you must give history manager an Object Node which generates events. This creates a table for all events from this object. I hope thats OK.

[zerox1212]

Started on being able to read events. Not quite finished, but I thought I would share the commits on my fork to make sure it looks OK for you guys. If I'm going totally in the wrong direction I don't want to waste my time.

https://github.com/zerox1212/python-opcua/commits/master

[oroulet]

Ja this sounds correct, call it source node. Events are generated by a
source node

On Wed, Apr 20, 2016, 03:37 Andrew notifications@github.com wrote:

I don't fully understand your changes, but I have the first half of events
history working (saves events, can't read them yet).

I made the interface so that you must give history manager an Object Node
which generates events. This creates a table for all events from this
object. I hope thats OK.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

Can you please explain how an EventResult gets populated with data? When I look at the class definition there isn't much there, but the object that shows up in history via the subscriptions is filled with data.

At the moment I don't know how to rebuild the Event for historical read.

Event class:

class EventResult():
    """
    To be sent to clients for every events from server
    """

    def __init__(self):
        self.server_handle = None

    def __str__(self):
        return "EventResult({})".format([str(k) + ":" + str(v) for k, v in self.__dict__.items()])
    __repr__ = __str__

Data that shows up in history:

[destogl]

EventResults are populated in event.py file

[oroulet]

you should not use the EventResult class. your method storage.read_node_history should return a list og EventFields constructed from the filter argument. This is rather complicated but you can have a look at internal_server.py and probably copy the code directly from there (if you can put code in a method and reuse it it is even better....

[oroulet]

Here is the method:

def _get_event_fields(self, evfilter, event):
    fields = []
    for sattr in evfilter.SelectClauses:
        try:
            if not sattr.BrowsePath:
                #val = getattr(event, ua.AttributeIdsInv[sattr.Attribute])
                val = getattr(event, sattr.Attribute.name)
                val = copy.deepcopy(val)
                fields.append(ua.Variant(val))
            else:
                name = sattr.BrowsePath[0].Name
                val = getattr(event, name)
                val = copy.deepcopy(val)
                fields.append(ua.Variant(val))
        except AttributeError:
            fields.append(ua.Variant())
    return fields

Maybe it should be moved to event.py as function

[zerox1212]

I understand now what needs to happen. Support for historizing custom events is going to be a big pain. I have started the implementation, but it's going to take some time before it's working.

[oroulet]

No it really should not be that complicated. Create a PR so we can see what you are doing and I can help you. You main issue is that you will have different attributes and you cannot have static columns. maybe store event values as json in one column for example.

[zerox1212]

It's messy at the moment so I'm not going to make a PR. You can dig around at https://github.com/zerox1212/python-opcua/commits/master if you want to see what I'm working at.

I read part 11 and part 4 of the spec, but I still don't know what object to return for storage.read_node_history. Returning a list of variant lists (fields) doesn't work. If I only return the fields themselves then how can I return multiple results? Do I need to return something like a UA EventNotificationList?

[oroulet]

Making a PR does not mean that I will merge it :-) but it makes it easy for
me and others to see your work. Can you do it?

On Fri, Apr 29, 2016, 07:36 Andrew notifications@github.com wrote:

It's messy at the moment so I'm not going to make a PR. You can dig around
at https://github.com/zerox1212/python-opcua/commits/master if you want
to see what I'm working at.

I read part 11 and part 4 of the spec, but I still don't know what object
to return for storage.read_node_history. Returning a list of variants
lists (fields) doesn't work. If I only return the fields themselves then
how can I return multiple results? Do I need to return something like a UA
EventNotificationList?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#129 (comment)

[zerox1212]

I will make a PR tonight because my fork is missing the merge of events by @destogl at the moment.

[zerox1212]

I'm very sleepy but I made a PR for you to review. :)
#171

[zerox1212]

My history is working well, I'm now trying to make tests.

Can someone explain why this (perhaps @destogl ):
if self.aspace.get_attribute_value(params.ItemToMonitor.NodeId, ua.AttributeIds.EventNotifier).Value.Value & 1 == 0:

Doesn't pass this test:

def test_subscribe_events_to_wrong_node(self):
        sub = self.opc.create_subscription(100, MySubHandler())
        with self.assertRaises(ua.UaStatusCodeError):
            handle = sub.subscribe_events(self.opc.get_node("i=85"))
        o = self.opc.get_objects_node()
        v = o.add_variable(3, 'VariableNoEventNofierAttribute', 4)
        with self.assertRaises(ua.UaStatusCodeError):
            handle = sub.subscribe_events(v)
        sub.delete()

The default code is this:
if self.aspace.get_attribute_value(params.ItemToMonitor.NodeId, ua.AttributeIds.EventNotifier).Value.Value != 1:

This will not work if the source node has bit 2 set, which is required for "event history read".

What I don't get is that during test test_subscribe_events_to_wrong_node it still returns correctly. I get ua.StatusCodes.BadServiceUnsupported but the test still fails.

I am just learning how tests work now, so I don't understand what is happening.

Thanks

[destogl]

@zerox1212 I am not sure what is happening here, but seams like you are missing bit operations with number operations, probably you should just stay by bits operations. I don't have a lot of experience using bit operations in python so I can not tell you exactly what is happening.

[zerox1212]

I solved it. part of that test causes the event notifier attribute to have a None type. Obviously you can't do bitwise & operator on a None type. I made a separate check for None before doing the bit operation and now it's working.

As far as I can tell, bit operations in python seem difficult to use. You can only really get a byte, not a bit array.

[zerox1212]

How can I initiate an event history read from server side?

In tests for reading history of data values the node has a method named read_raw_history:

res = self.var.read_raw_history(None, end, 3)

I don't see a method for server reading node event history. Do I need to make one?

[destogl]

If there is question for myself then: Yes. I am also not aware of existence of such a method

[oroulet]

The method read_raw_history is part of noise class isn't it? Then it is available on server and client. Or do I misunderstand something?

[zerox1212]

It is part of the Node class. This method is only for reading variable history, not event history. I tried for a few hours to make this method for events, but it isn't working yet.

[zerox1212]

@plieningerweb I don't know when we can merge my events history implementation, but if you want to test it on your application you can grab the events_history branch of my fork. Let me know if you find any problems with it, I spent many many hours on it.

[oroulet]

@destogl I check a prosys server and SourceName of an event is a string there. Not a ByteString and not a Localized nor a QualifiedName. The xml spec also says string, so we should us a string not a bytestring. fixing this

[oroulet]

scrutinizer is not happy with your classes :-) not really an issue but maybe see if you can split some methods..

[zerox1212]

I was already planning on doing this. Now that it's merged I can rebase and and try to make scrutinizer happier. Then when where clause is merged I will fix that for history as well. Thanks.

[zerox1212]

Scrutinizer stuff could still be improved, but I think this issue can be closed. If something comes up it will be better if it's a new issue.

[oroulet]

Events do not with any more with prosys client/server. This is something we
need to fix... But it might be a bit complicated..

On Fri, Jun 3, 2016, 05:25 Andrew notifications@github.com wrote:

Scrutinizer stuff could still be improved, but I think this issue can be
closed. If something comes up it will be better if it's a new issue.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#129 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ACcfzvXBO5YAREIcP_H11PdY6tJsJFzfks5qH569gaJpZM4HWeRX
.

[oroulet]

this now works. closing