Skip to content
A lightweight, HTTP-centric, log-based (Kafka-like) message queue.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
biglog
cmd/netlog
integration
transport
vendor
.travis.yml
CONTRIBUTING.md
LICENSE
README.md
errors.go
integrity_checker.go
integrity_checker_test.go
message.go
message_buffer.go
message_buffer_test.go
message_test.go
netlog.go
segment_monitor.go
streamer_atomicmap.go
topic.go
topic_atomicmap.go
topic_test.go
topicscanner.go
topicscanner_atomicmap.go
topicscanner_test.go
util.go
util_test.go

README.md

Build Status godoc reference

NetLog

A lightweight, HTTP-centric, log-based (Kafka-style) message queue.

Alpha software

This is still early software and potentially buggy. To peek at the internals start with BigLog.

Roadmap

  • low-level log management
  • HTTP transport
  • scanner based pub/sub
  • custom data retention policy
  • persistent scanners
  • batching
  • compression
  • good test coverage
  • proper documentation
  • streaming based pub/sub
  • async replication
  • kinesis-compatible transport
  • gRPC transport

Non-goals

  • Match Kafka's performance.
  • Distributed system.

Getting started

While posting and fetching single messages is very inefficient, it's the simplest way to get started using nothing but curl commands.

# compile server
go get github.com/ninibe/netlog/cmd/netlog

# run server
bin/netlog

# create new topic
curl -XPOST localhost:7200/demo

# post messages
curl -XPOST localhost:7200/demo/payload --data-binary "message number one"
curl -XPOST localhost:7200/demo/payload --data-binary "message number two"
curl -XPOST localhost:7200/demo/payload --data-binary "message number three"

# check topic info
curl localhost:7200/demo

# create scanner
curl -XPOST "localhost:7200/demo/scanner?from=0"

export SC="...UUID RETURNED..."

# start scanning...
curl -XGET "localhost:7200/demo/scan?id=$SC"
x times ...

# wait 5 seconds for new messages
curl -XGET "localhost:7200/demo/scan?id=$SC&wait=5s"

# wait 5 minutes
curl -XGET "localhost:7200/demo/scan?id=$SC&wait=5m"

# post more messages in another window
curl -XPOST localhost:7200/demo/payload --data-binary "message number four"
curl -XPOST localhost:7200/demo/payload --data-binary "message number five"

# new scanner since 1 minute ago
curl -XPOST "localhost:7200/demo/scanner?from=1m"

One-line-ish pub/sub

# create new topic
curl -XPOST localhost:7200/pubsubdemo

# get scanner ID with jq
export SCANNER=$(curl -s -XPOST "localhost:7200/pubsubdemo/scanner?from=0&persist=true" | jq -r .id)

# subscribe to the topic
while true; do; curl "localhost:7200/pubsubdemo/scan?id=$SCANNER&wait=1h" && echo; done

# IN ANOTHER WINDOW

# publish on the topic
while true; do; read data; curl localhost:7200/pubsubdemo/payload --data-binary $data; done
# write something and hit enter

Contributing

Contributions are more than welcome, check the contributing guidelines. To ask any questions you can write to the netlog-dev mailing list.

You can’t perform that action at this time.