TAXII 2.1 server in Golang.
- Golang 1.10 or 1.11
- SQLite
make
To run all tests: make test
"Helper" functions are in test_helper_test.go. The goal with this file was to put repetitive code that make the
tests verbose into a DRY'er format.
Security checking tool: https://github.com/securego/gosec
make sec to run security tests
make sec pkg=<package> to run against a specific package. Example: make sec pkg=sqlite to run it against sqlite package
Cabby uses sqlite3 as its data store. The library in golang being used requires C extensions/bindings and as such doesn't build on a Mac for a Ubuntu OS. Therefore vagrant vm is used to build a debian for ubuntu (i know...gross).
Building debian package for a vagrant VM running ubuntu: make build-debian
# tail the log for cabby
sudo journalctl -u cabby -f
# pipe it to less
sudo journalctl -u cabby | lessI used this project to also explore the TICK stack: https://www.influxdata.com/time-series-platform/
To use/play with it, run make vagrant
Syslog input reference (the VM uses syslog to as an input to influx-db):
- Example: https://fabianlee.org/2017/05/21/golang-running-a-go-binary-as-a-systemd-service-on-ubuntu-16-04/
- Prod config for linux: https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service#413408
- Debian Policy Manual: https://www.debian.org/doc/debian-policy/#debian-policy-manual
- Dependencies in Debian: https://www.debian.org/doc/debian-policy/#s-binarydeps
- Depend Differences: https://askubuntu.com/questions/83553/what-is-the-difference-between-dependencies-and-pre-depends#83559
- How to use systemctl: https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units
- Understanding systemd: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
Release branches are associated to a TAXII spec version and a SemVer version down to the minor revision.
Patches are annotated by creating a tag off of the release branch.
Examples:
- First major release of a TAXII 2.0 server:
- branch:
release/2.0/1.0 - tag:
release/2.0/1.0.0 - A version of TAXII 2.0 spec server that has backward incompatible changes BUT is still a TAXII 2.0 server
- branch:
release/2.0/2.0 - tag:
release/2.0/2.0.0 - Initial branch for a 2.1 spec server:
- branch:
release/2.1/0.0 - tag:
release/2.1/0.0.1(once a chunk of code worth tagging is ready)
The make task will generate certs and a default config file. Edit the config/cabby.json file to adjust things like
- port
- data store file path
- cert paths
Using Sqlite as a light-weight data store to run this in development mode. Goal is to move to some kind of JSON store (rethinkdb or elasticsearch) in the future. See below API examples for setup instructions.
The examples below require
- jq
- sqlite
On a mac you can install via brew:
brew install sqlite
brew install jqSet up the DB for dev/test:
make dev-db
The user set up with make dev-db is an admin (so it can do admin things like create/update/delete certain resources).
In another terminal, run a server:
make run
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/taxii2/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/taxii2/' | jq .# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/' | jq .# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/' | jq .
# view 1 of N
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/?limit=1' && echo# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/' | jq .In the above example, new collections were added. Kill the server (CTRL+C) and make run again. The logs will show new routes are added.
Now post a envelope of STIX 2.0 data:
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/malware_envelope.json | jq .From the above POST, you get a status object. You can query it from the server
export STATUSID=<your status id>
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' "https://localhost:1234/cabby_test_root/status/$STATUSID/" | jq .
unset STATUSID# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/manifest/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/manifest/' | jq .# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' | jq .
# view 1 of N
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?limit=1' && echo
# view 1 0f N parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?limit=1' | jq .Post an envelope with same objects but new versions to illustrate:
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/versions_envelope.json | jq .# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f/versions/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f/versions/' | jq .# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.stix+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad/' && echo
# parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.stix+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad/' | jq .# filter on types
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[type\]=indicator,malware' | jq .
# filter on id
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[id\]=indicator--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f' | jq .
# add objects to filter on versions
# the below envelope has objects that already exist; status will have 3 failures
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -H 'Content-Type: application/vnd.oasis.taxii+json' -X POST 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' -d @backends/sqlite/testdata/versions_envelope.json | jq .
# filter on latest versions (indicator will be 2018 provided data above has been added)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=last' | jq .
# filter on oldest versions (indicator will be 2016)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=first' | jq .
# filter on specific versions (indicator will be 2017)
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/?match\[version\]=2017-01-01T12:15:12.123Z' | jq .NOTE: These actions are destructive, use make dev-db to reset the dev db after deleting.
# with headers
curl -isk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -X DELETE 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad' && echo
# OR parsed json
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' -X DELETE 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/relationship--44298a74-ba52-4f0c-87a3-1824e67d7fad' | jq .
# view objects to verify
curl -sk -basic -u test@cabby.com:test-password -H 'Accept: application/vnd.oasis.taxii+json' 'https://localhost:1234/cabby_test_root/collections/352abc04-a474-4e22-9f4d-944ca508e68c/objects/' | jq .