Queue based data collector, transmitter with store-and-forward. Useful for online/offline data collection, back pressure buffering or general queuing. REST / HTTP post.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dockerfiles
rtq
vendor conversion to glide Nov 2, 2018
.gitignore
.travis.yml
LICENSE Initial commit Apr 11, 2018
README.md
glide.lock
glide.yaml conversion to glide Nov 2, 2018
goreleaser.yml updated homebrew tap Apr 25, 2018
mast.jpg
rxtx.go logger sync error message Nov 2, 2018

README.md

rxtx data transmission irsync Release Build Status Go Report Card Maintainability GoDoc FOSSA Status

Docker Container Image Size Docker Container Layers Docker Container Pulls

rxtx

rxtx is a queue based data collector > data transmitter. Useful for online/offline data collection, back pressure buffering or general queuing. rxtx uses bbolt maintained by CoreOs, a single file database for storing messages before they can be sent.

rtbeat was developed to consume rxtx POST data and publish as events into elasticsearch, logstash, kafka, redis or directly to log files.

Test on MacOs

Install with brew

brew tap txn2/homebrew-tap
brew install rxtx

Help

rxtx -h

Test Docker Container

Help

docker run --rm -it txn2/rxtx -h

on arm 6/7 based device:

docker run --rm -it txn2/rxtx:arm32v6-1.2.0 -h

Test Source

Help

go run ./rxtx.go -h

Usage of rxtx:
  -batch int
        Batch size. (default 5000)
  -ingest string
        Ingest server. (default "http://localhost:8081/in")
  -interval int
        Seconds between intervals. (default 30)
  -maxq int
        Max number of message in queue. (default 2000000)
  -name string
        Service name. (default "rxtx")
  -path string
        Directory to store database. (default "./")
  -port string
        Server port. (default "8080")

Start server on 8080

go run ./rxtx.go 

Add message to queue

The rxtx services accepts HTTP POST data to an API endpoint in the following form /rx/PRODUCER/KEY/LABEL/.../. One label is required, however as many labels as necessary may be added, separated by a forward slash.

curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data

Add message to queue every second

 while true; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; sleep 1; done

Add 1000 messages to the queue.

 time for i in {1..1000}; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; done

Building and Releasing

rxtx uses GORELEASER to build binaries and Docker containers.

Test Release Steps

Install GORELEASER with brew (MacOS):

brew install goreleaser/tap/goreleaser

Build without releasing:

goreleaser --skip-publish --rm-dist --skip-validate

Release Steps

  • Commit latest changes
  • Tag a version git tag -a v2.0 -m "Version 2.0"
  • Push tag git push origin v2.0
  • Run: GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist

Resources

License

FOSSA Status