Membership operator doesn't work as expected for Var expressions

[Onager]

This looks to me like two issues:

• expressions that are values that contain strings don't trigger the string membership logic
• expression values that are unicode strings don't trigger the string membership logic as six.strings_types contains only str, and not unicode.

Debugging values from breakpoint at

dotty/efilter/transforms/solve.py

Line 775 in ee0f320

return Result(needle in values, ())

expr = {Membership} Membership(\n Literal('a message')\n Var('body'))
 _BinaryExpression__abstract = {bool} True
 _Expression__abstract = {bool} True
 arity = {int} 2
 children = {tuple} <type 'tuple'>: (Literal('a message'), Var('body'))
 element = {Literal} Literal('a message')
 end = {int} 25
 lhs = {Literal} Literal('a message')
 return_signature = {ABCMeta} <class 'efilter.protocols.boolean.IBoolean'>
 rhs = {Var} Var('body')
 set = {Var} Var('body')
  _Expression__abstract = {bool} True
  _ValueExpression__abstract = {bool} True
  arity = {int} 1
  children = {tuple} <type 'tuple'>: ('body',)
  end = {int} 4
  return_signature = {type} <class 'efilter.protocol.AnyType'>
  source = {str} 'body contains \\'a message\\''
  start = {int} 0
  type_signature = {tuple} <type 'tuple'>: (<type 'basestring'>,)
  value = {str} 'body'
 source = {str} 'body contains \\'a message\\''
 start = {int} 0
 type_signature = {tuple} <type 'tuple'>: (<class 'efilter.protocols.eq.IEq'>, <class 'efilter.protocols.iset.ISet'>)
needle = {str} 'a message'
values = {generator} <generator object __solve_and_destructure_repeated at 0x7fe778690550>
 gi_code = {code} <code object __solve_and_destructure_repeated at 0x7fe7944a81b0, file "/usr/lib/python2.7/dist-packages/efilter/transforms/solve.py", line 146>
 gi_frame = {frame} __solve_and_destructure_repeated [solve.py:146]  id:49887728
 gi_running = {int} 0
vars = {ScopeStack} ScopeStack(LibraryModule(name='stdcore', vars={'materialize': <efilter.stdlib.core.Materialize object at 0x7fe7944d2550>, 'singleton': <efilter.stdlib.core.SingletonReducer object at 0x7fe7944d24d0>, 'int': <type 'int'>, 'float': <type 'float'>, 'find': <e
 globals = {LibraryModule} LibraryModule(name='stdcore', vars={'materialize': <efilter.stdlib.core.Materialize object at 0x7fe7944d2550>, 'singleton': <efilter.stdlib.core.SingletonReducer object at 0x7fe7944d24d0>, 'int': <type 'int'>, 'float': <type 'float'>, 'find': <efilter.stdl
 locals = {TestEvtRecordEvent} <tests.analysis.tagging.TestEvtRecordEvent object at 0x7fe778684290>
  COMPARE_EXCLUDE = {frozenset} frozenset([u'store_number', u'display_name', u'uuid', u'data_type', u'timestamp', u'filename', u'store_index', u'tag', u'pathspec', u'inode'])
  CONTAINER_TYPE = {unicode} u'event'
  DATA_TYPE = {unicode} u'windows:evt:record'
  body = {unicode} u'this is a message'
  data_type = {unicode} u'windows:evt:record'
  display_name = {NoneType} None
  event_identifier = {int} 16
  filename = {NoneType} None
  hostname = {NoneType} None
  inode = {NoneType} None
  offset = {NoneType} None
  pathspec = {NoneType} None
  source_name = {unicode} u'Messaging'
  store_index = {NoneType} None
  store_number = {NoneType} None
  tag = {NoneType} None
  timestamp = {int} 1464181206000000
  uuid = {unicode} u'c3f4974cacc44430b8a1b63d17308e90'
 scopes = {list} <type 'list'>: [LibraryModule(name='stdcore', vars={'materialize': <efilter.stdlib.core.Materialize object at 0x7fe7944d2550>, 'singleton': <efilter.stdlib.core.SingletonReducer object at 0x7fe7944d24d0>, 'int': <type 'int'>, 'float': <type 'float'>, 'find

[the80srobot]

Hmm, six.string_types should contain basestring. I added a test case and unicode works.

I apologize, but I actually don't understand the blob of text - can you tell me what exact error you're getting?

[Onager]

I'll update this a little more when I have some more time, but in the interim:

I'm not sure what I was seeing to think that six.string_types was just str in Python2, you're quite right that this isn't an issue.

In the debug output above, the expression is Membership(\n Literal('a message')\n Var('body')) which should match the body var, whose value is "this is a message". However, this breakpoint was set at

dotty/efilter/transforms/solve.py

Line 775 in ee0f320

return Result(needle in values, ())

so the function hasn't returned at

dotty/efilter/transforms/solve.py

Line 772 in ee0f320

return Result(needle in expr.set.value, ())

like I think it it should have.

I suspect the issue is at

dotty/efilter/transforms/solve.py

Line 770 in ee0f320

if (isinstance(expr.set, ast.Literal)

as expr.set is Var not ast.Literal in the debug values.

Hopefully that clear things up a little. I'll try and expand a little more when I have a bit more time. log2timeline/plaso#899 is the parent issue from Plaso where we encountered this.

[Onager]

And I didn't actually answer your question: the error is that an objectfilter expression like: `body contains 'a message'`` doesn't match when run with efilter.api.apply(expression, vars=event) and event has an attribute "body" that contains a string "this is a message", and event has a protocol definition that looks like this: https://github.com/log2timeline/plaso/blob/master/plaso/containers/interface.py#L58.

That wasn't particularly eloquent, my apologies.

[Onager]

OK, failing test.
This event: https://github.com/Onager/plaso/blob/fix_tagging/tests/analysis/tagging.py#L56

Is not matched by this rule: https://github.com/Onager/plaso/blob/fix_tagging/test_data/test_tag_file.txt#L14

When it should be.

[the80srobot]

I think I finally figured out what's wrong and fixed it: https://codereview.appspot.com/302640043/

[the80srobot]

Fixed in 1.5