Support: Issues Adding TAXII Server #8832
Comments
zhantt1
added
needs triage
|
Seeing this same issue with the baseurl field limited to only 11 characters, including after upgrade to v2.4.168 EDIT: I don't know enough about PHP but I did find reference in app/Model/AppModel.php (line 1913)
|
|
Has anyone used this feature before? |
adulau
added
T: bug
|
@iglocska just confirmed the bug. |
|
Fixed on 2.4, make sure you git pull and the DB change should be applied automatically. |
|
Looks like that did the trick, baseurl is now able to be filled out correctly. I think fixing this bug has lead to #8941 however. |
|
I've updated to 2.4.168 and confirmed that the base URL can be filled. However, the dropdowns for API root and Collection are still empty. Does anyone know how to set those fields? |
|
@zhantt1 Hey did you ever get an answer for this, because I cant fill out API root or Collections either |
|
@ArchieCrawford1 unfortunately, I haven't heard anything. |
|
I didn't hear either |
|
We have the same Issue. |
|
this issue was introduced a while ago, and it doesn't seem like upgrading problematic versions to fixed versions fixes the db schema, and leaves the baseurl column in the taxii_servers table as int(11). if you upgrade your misp to the most recent version (2.4.170) and go to diagnostics, you will see that it dislikes the taxii_servers table's baseurl column. it will tell you that running something like the following will fix it:
this does not appear to occur automatically when upgrading, or by running the cake console updatedb or whatever command, so if you log into your misp db and run the above, it should fix it for you. i imagine that the project will eventually fix this, but in the meantime, this works. |
|
Thanks for the tip @dogmachine. Looks like I still needed to run that command when updating MISP to 2.4.171. Unfortunately, I'm still seeing empty dropdowns in the API root and collection fields. Since I can't set those, I don't see a way to connect to the TAXII 2.1 server in my environment. |
|
CIRCL have provided an article to help out https://www.misp-project.org/2023/04/29/MISP.how.to.push.to.a.taxii.server.html/ which specifies using medallion to create your Taxii 2.1 server. I've noticed in my troubleshooting that MISP is very particular about the URL it searches for to find the API root and collections, it tries to go to https://example.com/api/ while I had an opentaxii setup that was using https://example.com/services/ - I ended up having to move away from opentaxii's implementation and towards medallion (see: https://github.com/oasis-open/cti-taxii-server). |
|
@zhantt1 as @Byhird mentioned the config of these is a bit funny. you basically need to put the base url of the taxii2.1 service in, like https://taxii2.blah.com/ and not anything like https://taxii2.blah.com/taxii2/ (it is expecting the path where it will be able to find /taxii2/) or anything you might expect from reading the taxii2.1 spec. once you have entered that (with appropriate port if not uri schema conforming) you can click the spanner icons and the drop down fields will populate. it'd be nice if you could just type them in too, which is how everything else in misp works. even if you could type them in though, the taxii2.1 spec says api roots can be full urls or relative paths, and what misp records is neither. needs a bit of work making the language used less ambiguous. |
|
also it appears the collection id isn't stored in the db anywhere that i can see. there isn't a collection column in the taxii_server table, and the collection id appears to be hard coded as mentioned here |
|
Thanks @Byhird and @UFOSmuggler, your comments were helpful. I set up a medallion instance (using the memory backend) and confirmed that the API roots and collections appeared when I click on the spanner icon. Unfortunately, the "push to TAXII" button does not seem to work. When I press the button, the UI displays "TAXII push initiated", but the medallion console does not show any new API requests. I updated the medallion config to include a collection with the hardcoded collection ID @UFOSmuggler linked under the API root I was pushing to, but this did not work either (the collection remained empty). I have a few sample events and attributes in MISP that I expected would be pushed. Has anyone had any success pushing to their TAXII server? Also is there a feature to poll from the TAXII server? |
|
I tried pushing a simple MISP event into an OpenTAXII instance without success. In my setup, MISP event is imported into MISP using MISP in-built STIX importer and tried to export that into the OpenTAXII instance. Based on @UFOSmuggler's findings I went and hard-coded the API root (tested both |
|
@dragsu I checked my MISP logs and also see those messages when I attempt to push to my medallion server. |
|
The further I dig into this the more I am convinced that the wrong code was merged or something. I don't think it could have worked for anyone. |
|
Interesting, we'll have have a look with @chrisr3d, we only tried pushing to medallion, that seems to have worked, but it's super early days for the taxii push, so your mileage may vary. |
|
Hi @iglocska and @chrisr3d, thanks for taking a look into this. This makes me think it cannot have worked and must be the wrong code: To create a taxii push job, the push_taxii command calls the taxii push function with the wrong arguments. I'm guessing one of these is meant to be different. If I fix this, it then seems the taxii server filter arguments don't evaluate correctly and every event in my MISP instance is sent to push (though fails to do so, presumably hits execution timeout). I don't yet know why that is though. Other than that, as mentioned above there doesn't appear to be anywhere in the DB or app model to store the taxii collection uuid you want to push to, and it seems to be hard coded. I'm not great at PHP though, so I could be misunderstanding what is going on there. Apologies if this is the case. |
|
No worries, high chance those are correct observations and we've accidentally left something hard-coded in there that worked for us. We'll revise it today and the fixes will be in the next release. |
|
I've pushed a first fix for testing, indeed it was broken as hell. Still rolling with a hard-coded collection, will change that this afternoon if all goes right, some meetings first though... :) |
|
Hello everyone; I'm having issues with Taxii servers connected with MISP. Then I switched to Medallion and succeeded in creating a Taxii server and connecting to MISP, but I'm having issues pushing data to the Taxii server. Do anyone had and solved the same problem? |
|
hi @pdelv if you were using cabby then i am assuming your opentaxii instance was only configured for taxii 1.x (which misp does not support) as cabby only supports taxii 1.x is there any chance you could share the misp event that you are attempting to push to the medallion server? |
|
Dear @UFOSmuggler, |
|
g'day @pdelv i ingested your (rather strange) misp event and successfully pushed it to my medallion instance: i am wondering if your stix libraries are not the misp project forked ones. try this: |
|
@dragsu I configured my OpenTAXII instance for TAXII 2 by following your thread here eclecticiq/OpenTAXII#247 and am using a fixed version of OpenTAXII (andrewbeard/OpenTAXII@2d135dd) to bypass the UUID encoding bug. I am able to query my OpenTAXII API root from MISP, but cannot query the collection (dropdown is empty). I know the collection exists, because I can query it using curl. Do you have any ideas on what could be wrong? |
|
hi @zhantt1, can you please show me an example of how you can curl the collection? this will help me figure out whether or not your "api root" as configured in the screenshot is correct or not. i expect that it is just wanting http://192.168.3.79:9000/ rather than a uuid and whatever follows. feel free to redact stuff from your curl example. |
|
@zhantt1 is your If you want to do more debugging you can play with the Python script separately. It can be found under E.g. @UFOSmuggler Even though API Root field is showing the entire URI, under the hood I think it only tracks the API Root id, which is a UUID. |
@dragsu api roots are just strings or uris. you could choose to have one as a uuid, but usually they are meant to be a human readable thematic grouping of taxii resources. additionally, /taxii2 is the discovery endpoint that tells you where and what the api roots are. it MUST be at /taxii2. misp takes the base url, searches for a discovery service on /taxii2, uses it to populate the api root dropdown which in turn populates the collection dropdown. I enter http://192.168.5.100:5000 (a locally running dockerised medallion taxii 2.1 server) into misp, add my "api key" (base64 credentials in this case), and hit the api root wrench, medallion returns: It now knows the api roots, and i select "trustgroup1" from the dropdown. i now click the collections wrench, medallion returns: It now knows the collections, and i can select one from the dropdown and hit save. edit: collection ids are uuids though, which may be what you're thinking about. for completeness, here is a curl conversation also reflecting the above: At no point do we see uuids for api roots, only collection ids. further, here is what is in the taxii_servers table: i believe this id and uuid is for misp purposes rather than anything taxii related and simply reflects the set of configuration options chosen for that specific taxii server. my comment to the other user was noting a uuid as their api root, which i doubt should be there, which is why i asked to see a curl example. |
|
@UFOSmuggler Thanks for the explanation. My curl command for API discovery returns API roots as a collection of |
|
@dragsu that sounds like a possibly misconfigured opentaxii instance? there shouldn't be anything under /taxii2/ unless you've specified it as an api root so its not just the discovery endpoint (unsure if opentaxii would even allow this)? it is possible but generally unusual that you'd have api roots that are literally uuids. could you share your opentaxii config? feel free to redact stuff |
|
@dragsu actually the spec does allow for api roots to be under taxii2, nothing explicitly forbids it and they even show it as an example of a valid api root, my apologies. |
|
Just for completeness, I have posted the config at eclecticiq/OpenTAXII#247 (comment). |
|
@UFOSmuggler here is my curl command and response. |
|
@dragsu I have been experimenting with taxii_push.py without success. When I use the baseURL I made potential progress with the baseURL This was the same MISP event that I successfully pushed to Medallion (an event with an IP attribute). I exported the event to JSON and removed the "response" field around the event: |
|
i've fired up opentaxii again and set up some taxii 2.1 resources. it appears that all api roots being under /taxii2/ and all api roots being uuids is a common opentaxii taxii 2.1 pattern that i had somehow forgotten about. so in both of your cases, you appear to have bog standard opentaxii looking configurations, apologies for my earlier skepticism. however, i have discovered that misp is incorrectly dealing with the api roots presented by opentaxii. if i ask opentaxii for its api roots i get: as you can see, the api root is /taxii2/blah. when I add this server to MISP, it chooses to omit the /taxii2/ from the api root, and makes http requests to /1cbd7c30-48ea-431e-8d76-c611bc6f27a4/ instead: obviously the opentaxii server doesn't like this, and we cannot populate the collections dropdown as a result. someone appears to have noticed this already in this issue. |
|
@zhantt1 Try using Also, I did a quick test with @UFOSmuggler that aligns with what I observed in my previous testing. I'm not sure whether it is something MISP needs to fix or OpenTAXII. As far as I know, there is no one with write access to OpenTAXII repo. I briefly looked at OpenTAXII code (snippet below) and I think it needs the |
@dragsu it's misp's taxii implementation not conforming to the taxii 2.1 spec, so misp need to fix it. shouldn't be a huge change. you could probably force all the correct data into the taxii_servers table directly and get it to work today. |
|
@UFOSmuggler thanks for digging further. @dragsu Your workaround for setting the collection worked, thanks. I then attempted to push data (an event with an IP attribute) to OpenTAXII via the UI. I did not notice any errors in MISP's error.log. Then, I tried using the taxii_push.py script. I updated the event JSON by removing the "Event" field as you recommended. The script returns an error and the same error message appeared on the OpenTAXII side as before. My guess is that the MISP to STIX translation does not create a "modified" field leading to the KeyError('modified') on the OpenTAXII side. When I attempted to push a STIX object without that field to OpenTAXII via curl, OpenTAXII returned error 400 saying it was invalid STIX. The STIX spec itself says the "modified" field is optional. |
|
Which version of |
|
@dragsu based on the readme's badge link in |
|
@zhantt1 If you can drop your payload, I can give it a quick test. |
|
@dragsu thanks, here is the MISP data I tried to use for taxii_push.py |
|
@zhantt1 sorry for the late reply. I have successfully pushed the data into my TAXII server using On a different note, I reckon it will be very handy to have STIX collection management capability as part of the TAXII Servers configuration in MISP. |
|
@dragsu sounds like there is a problem somewhere with my set up. Can you tell me how to update misp-stix to the latest version on a MISP instance? Thanks. |
|
@zhantt1 If there are any library issues, they should be written to one of the error logs ( Apart from that I had to pip install, pytz, simplejson, stix2-patterns, ordered_set, lxml, mixbox, taxii2-client, stix, and stix2. |
|
Update, I was troubleshooting today against MISP 2.4.173. I confirmed that STIX objects were being received on the Open TAXII side (identity, observed data, and file) by adding a print statement in their code. However, OpenTAXII for some reason was not able to properly ingest them into the collection (same KeyError as before). Then I updated MISP to 2.4.174 and now get an error on the MISP side in exec-errors.log: I think I saw that error in the thread @UFOSmuggler linked, but haven't looked deeply. |
Dear @UFOSmuggler thanks a lot for your answer, and sorry for the late reply, but it was a really dense period. I wanted to give you positive feedback that I managed my TAXII issues before writing you back. May I ask for some help in figuring out how to prepare this curl to be sent to Medallion? I'm sorry if my question could look so noob but i'm not that experienced in this field. P.s. One example curl I used: |
Update: I reverted back to 2.4.173 and was able to push an md5 attribute to medallion. However, exec-errors.log reports errors: |
|
Update: I installed MISP 2.4.174 on a different machine and was able to push to medallion after I manually installed the python package taxii2-client. Unfortunately, my opentaxii instance still does not like the data MISP 2.4.174 is sending 🤷♂️ |
Support Questions
I am attempting to link an OpenTAXII server to a MISP instance under Sync Actions > List TAXII Servers.
When I click "Add TaxiiServer", I experienced problems with some fields.
I inputted the name, owner, and baseurl of the TAXII server and pressed Submit. The web dev console printed a 500 Internal Server error.
I've attached images of the baseurl field being restricted to 11 characters, the empty dropdown from the Api Root field, and the popup error after pressing Submit.
Thanks for any insight.
MISP version
2.4.166
Operating System
Ubuntu
Operating System version
22.04
PHP version
7.4
Browser
Firefox
Browser version
No response
Relevant log output
No response
Extra attachments
Code of Conduct
The text was updated successfully, but these errors were encountered: