events with No attributes #11
Comments
|
https://github.com/MISP/MISP-Taxii-Server/blob/master/misp_taxii_hooks/hooks.py#L66 We only add the event if it has more than 1 attrib. Already implemented. Have not been able to replicate. |
|
to replicate, you can try to pull the hailataxii feed (cybertracker) out of the 3.7K events it would create few without an attribute. |
|
I also encountered this when testing with STIX samples from https://stix.mitre.org/language/version1.1.1/samples.html |
|
Ohhhhh, I hate STIX so much. They're array values to indicators. ARRAY VALUES. I want to die. |
|
MISP rejects them as being invalid, so you get the event with 0 attribs |
|
Ok that should fix it. Pushed changed to PyMISP, MISP-STIX-Converter and this repo |
|
They weren't just array values They were
Haha kill me |
|
5 days and nothing else posted, assuming fixed Just tell me if it isn't it should be |
|
Hi @FloatingGhost, {"logger": "opentaxii.middleware", "exception": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 104, in load_stix\n stix_package = STIXPackage.from_json(data)\n File \"/usr/local/lib/python3.4/dist-packages/mixbox/entities.py\", line 486, in from_json\n d = json.loads(json_doc)\n File \"/usr/lib/python3.4/json/__init__.py\", line 318, in loads\n return _default_decoder.decode(s)\n File \"/usr/lib/python3.4/json/decoder.py\", line 343, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/usr/lib/python3.4/json/decoder.py\", line 361, in raw_decode\n raise ValueError(errmsg(\"Expecting value\", s, err.value)) from None\nValueError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 110, in load_stix\n stix_package = STIXPackage.from_xml(stix)\n File \"/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py\", line 249, in from_xml\n return entity_parser.parse_xml(xml_file, encoding=encoding)\n File \"/usr/local/lib/python3.4/dist-packages/mixbox/parser.py\", line 179, in parse_xml\n xml_etree = get_etree(xml_file, encoding=encoding)\n File \"/usr/local/lib/python3.4/dist-packages/mixbox/xml.py\", line 55, in get_etree\n return etree.parse(doc, parser=parser)\n File \"src/lxml/lxml.etree.pyx\", line 3442, in lxml.etree.parse (src/lxml/lxml.etree.c:81716)\n File \"src/lxml/parser.pxi\", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118903)\n File \"src/lxml/parser.pxi\", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:119186)\n File \"src/lxml/parser.pxi\", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117974)\n File \"src/lxml/parser.pxi\", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:112701)\n File \"src/lxml/parser.pxi\", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105896)\n File \"src/lxml/parser.pxi\", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:107604)\n File \"src/lxml/parser.pxi\", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:106458)\n File \"<string>\", line 5\nlxml.etree.XMLSyntaxError: Opening and ending tag mismatch: Handling line 3 and Handling, line 5, column 34\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py\", line 83, in process\n response_message = handler.handle_message(self, message)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 126, in handle_message\n return InboxMessage11Handler.handle_message(service, request)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 65, in handle_message\n inbox_message_id=inbox_message.id if inbox_message else None)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py\", line 164, in create_content\n collection_ids=collection_ids, service_id=service_id)\n File \"/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py\", line 267, in send\n for receiver in self.receivers_for(sender)]\n File \"/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py\", line 267, in <listcomp>\n for receiver in self.receivers_for(sender)]\n File \"/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py\", line 62, in post_stix\n package = pymisp.tools.stix.load_stix(content_block.content)\n File \"/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py\", line 17, in load_stix\n stix = convert.load_stix(stix)\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 136, in load_stix\n return load_stix(f)\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 113, in load_stix\n raise STIXLoadError(\"Could not load stix file. {}\".format(ex))\nmisp_stix_converter.errors.STIXLoadError: Could not load stix file. Opening and ending tag mismatch: Handling line 3 and Handling, line 5, column 34 (<string>, line 5)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.4/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/local/lib/python3.4/dist-packages/flask/app.py\", line 1598, in dispatch_request\n return self.view_functions[rule.endpoint](**req.view_args)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py\", line 76, in wrapper\n return _process_with_service(service)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py\", line 154, in _process_with_service\n response_message = service.process(request.headers, taxii_message)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py\", line 89, in process\n in_response_to=message.message_id)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/exceptions.py\", line 48, in raise_failure\n tb=tb)\n File \"/usr/local/lib/python3.4/dist-packages/six.py\", line 685, in reraise\n raise value.with_traceback(tb)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py\", line 83, in process\n response_message = handler.handle_message(self, message)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 126, in handle_message\n return InboxMessage11Handler.handle_message(service, request)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 65, in handle_message\n inbox_message_id=inbox_message.id if inbox_message else None)\n File \"/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py\", line 164, in create_content\n collection_ids=collection_ids, service_id=service_id)\n File \"/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py\", line 267, in send\n for receiver in self.receivers_for(sender)]\n File \"/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py\", line 267, in <listcomp>\n for receiver in self.receivers_for(sender)]\n File \"/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py\", line 62, in post_stix\n package = pymisp.tools.stix.load_stix(content_block.content)\n File \"/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py\", line 17, in load_stix\n stix = convert.load_stix(stix)\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 136, in load_stix\n return load_stix(f)\n File \"/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py\", line 113, in load_stix\n raise STIXLoadError(\"Could not load stix file. {}\".format(ex))\nopentaxii.taxii.exceptions.FailureStatus: Could not load stix file. Opening and ending tag mismatch: Handling line 3 and Handling, line 5, column 34 (<string>, line 5)", "event": "Status exception", "timestamp": "2017-06-28T13:49:46.116519Z", "level": "warning"}[FloatingGhost Edit] This is the XML block which causes the error I updated MISP-TAXII-Server, Stix-Converter, PyMISP. The XML seems to be valid. Thanks |
|
Seems like it all works, no attrs besides the original document, but that's expected behaviour I may have edited over your edit whilst extracting the error :P |
|
UPDATE: Update another time this repo, now it doesn't crash but any event is being created, got the same error plus this one: |
|
heh, seems my regex was a little hungry. Lemme satiate it a bit. |
FloatingGhost
added a commit
to MISP/MISP-STIX-Converter
that referenced
this issue
Jun 28, 2017
|
Try that! Pushed an update to the converter |
|
Hi! |
|
Then I cannot replicate. It works here and passes all tests. |
|
Your XML sample from above was used in a test. It passes just fine. |
|
This is what I see I'm going to print the content block so I can paste it here |
|
This are the XML Blocks I'm trying to find if I did something wrong with the DB or I didn't update everything |
|
Ok, try again. It should log more this time, and I think I fixed your issue along the way |
|
Tried, now a few of events are being created, for example 10/250, I think there is some problem yet. This is the output |
|
There we go, that'll update will log EVERYTHING and explain it |
|
Also, I can notice many events created without attributes |
|
Ok, I have another example.. I checked for that value on MISP and it's not present. If you want I can print the XML blocks again. |
|
That log is incomplete. After we print "Checking for existence..." it'll either say if it's unique or a duplicate |
|
Like so |
|
I attached a screenshot, this is what I see |
|
Working as intended. No issues to see there. |
|
Yes, but if I search for the value "updateceb.zapto.org" on MISP I won't find anything, is this normal? |
|
You might have updateceb.zapto.org/some_subpath MISP has no way to do exact search, it'll so substring though. |
|
If you run a pymisp |
|
Well, I have only 14 events and I searched for it manually one by one and it's not present |
|
|
|
Run the pymisp search. That'll tell you what's up |
|
Found it, thanks! It's not shown probably because under the column there is this error: |
|
Hi @FloatingGhost, At the moment I'm trying to understand which of those events are being inserted in MISP without attributes. |
|
Ok here we go.. I looked at the logs and I found an example of error I get EDIT: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I noticed that some events (specifically from hailataxii) are still producing events with empty attributes as the attributes is not formatted properly.
I am sure that the feeds should normalized at the source but I thought that it's not a bad idea to do a check after the adding just incase.
Note: If yout think this is useful I can request pull.
Regards,
MAstarux
The text was updated successfully, but these errors were encountered: