null not working in q/mq filter in subscriptions

[fgalan]

Situation:

Having this subscription in place:

   {
        "id": "59d675c5b473c61d5aa876f3",
        "notification": {
            "attrs": [
                "t"
            ],
            "attrsFormat": "normalized",
            "http": {
                "url": "http://10.x.y.z:10031/notify"
            }
        },
        "status": "active",
        "subject": {
            "condition": {
                "attrs": [
                    "t"
                ],
                "expression": {
                    "q": "t==null"
                }
            },
            "entities": [
                {
                    "idPattern": ".*"
                }
            ]
        }
    }

And creating this entity:

{
        "id": "E1",
        "type": "T",
        "t": {
            "metadata": {},
            "type": "StructuredData",
            "value": null
        }
}

Bug:

• Creating that entity doesn't triggers notification (i.e. timesSent is not present in subscription). Changing from/to null not triggers notification.

[fgalan]

This is related with functionality developed in #2359. Looking to cases/2359... the null_values_in_string_filters.test doesn't covers any subscription case.

[fgalan]

It seems the same problem occurs with mq and metadata filters.

[kzangeli]

First impression:
we probably don't distinguish between 'not present' and 'present but with NULL value'.
To distinguish between the two cases we'd need a boolean field 'present in payload', which I believe we don't have, unfortunately ...

Also first impression: Shouldn't be too hard to fix this problem.

[fgalan]

I don't remember this pretty well, but I think we implemented a type selector for attributes and metadata:

typedef enum ValueType
{
  ValueTypeUnknown,
  ValueTypeString,
  ValueTypeNumber,
  ValueTypeBoolean,
  ValueTypeVector,
  ValueTypeObject,
  ValueTypeNone
} ValueType;

ValueTypeNone is the one used for null.

Although maybe the lack of distintion between "not present" and "present but with NULL value" you refer is in the filter object, not in the attribute/metadata value, and this comment is pointless :)

[fgalan]

Could be related with #2587

[fgalan]

As discussed at skype, maybe is a matter of going from:

typedef enum ValueType
{
  ValueTypeUnknown,
  ValueTypeString,
  ValueTypeNumber,
  ValueTypeBoolean,
  ValueTypeVector,
  ValueTypeObject,
  ValueTypeNone
} ValueType;

to

typedef enum ValueType
{
  ValueTypeUnknown,
  ValueTypeString,
  ValueTypeNumber,
  ValueTypeBoolean,
  ValueTypeVector,
  ValueTypeObject,
  ValueTypeNull,      // old ValueTypeNone
  ValueTypeNotGiven   // new, to mean cases in which the value was not provided
} ValueType;

In order to resolve currenta ambiguety with ValueTypeNone to deal with both situations in the code.

[fgalan]

Fixed in PR #3039

However, I'm not going to close the issue yet. The fix is that PR is not definitive and I'll like to have a look to how this stuff is implemented (I'm assigning to me).

[fgalan]

Refactor done in PR #3046. That PR removes the #if temporal solution. Thus, I understand that PR provides a final solution and this issue could be closed.