This collector meant to be a highly available RESTful web service that receives events from client devices and secures them agnostic of cloud targets.
The collector is implemented in Go, runs on Ubuntu and OSX.
Events are sent in Protobuf (recommended) or JSON messages.
Currently supported cloud targets are (tested throughput on a c3.xlarge computer with 4vCPU in AWS using protobuf):
- Amazon Web Services Simple Notification Service: 59k events/minute, 70 multi payload requests/s
- Amazon Web Services Simple Storage Service: 6.2M events/minute, 8k payload requests/s
- Microsoft Azure Blob Storage: 6.05M events/minute, 7.8k multi payload requests/s
- Microsoft Azure Queue Storage: 5k events/minute, 5 multi payload requests/s
6Wunderkinder used a similar node.js based service that secured messages in AWS SNS. Based on experiences we've rewritten the app in Go that can handle 20x more requests on equal hardware resources.
Inspired by UNIX philosophy (do one thing and do it well) and Marcio Castilho's approach.
No official client is available at the moment. If you want to write your own please check out our pseudo client specification.
$ sudo make install/go && source ~/.profile # you can install golang with this on OSX/Ubuntu if you need it $ sudo make install/protobuf # initialize communication format $ make install/pkg # golang dependencies $ make install/symlink # if you want to use this package outside of $GOPATH $ sudo make install/wrk # install http benchmarking tool $ make install/utils # utils for development
After the package installation, please create your configuration file based on the sample configuration. You can also generate a configuration file with the following command:
$ make setup
Set up your environment variables.
export HAMUSTRO_CONFIG="config/yourconfig.json" export HAMUSTRO_HOST="localhost" export HAMUSTRO_PORT="8080"
You can start the server for development with the following command:
$ make dev
In the development mode it provides useful messages to track what's happening within the collector. Furthermore it notifies the clients with JSON responses on error.
To turn off the notifications and run the collector for production, please use the following command:
$ make server
You can run the unit tests with
$ make tests/run
You can send a single message to the server with
$ make tests/send/protobuf
If you want to start a stress test, please use
$ make tests/protobuf/n # 1-25 payloads/protobuf request $ make tests/protobuf/1 # 1 payload/protobuf request
Remember, you can use
json instead of
protobuf if you want. Using JSON is 50% slower though.
During the stress test, you can profile the heap/cpu/goroutine usage easily in development mode, just type
$ make profile/heap $ make profile/cpu $ make profile/goroutine
Copyright © 2016, Microsoft
Distributed under the MIT License.