A simple HTTP server for queuing lines of text
Clone or download
Latest commit f4d698a Mar 26, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
resources Initial commit Mar 25, 2017
src/lq Initial commit Mar 25, 2017
test/lq Initial commit Mar 25, 2017
.gitignore Initial commit Mar 25, 2017
LICENSE Initial commit Mar 25, 2017
Makefile Initial commit Mar 25, 2017
README.md Initial commit Mar 25, 2017
project.clj Initial commit Mar 25, 2017

README.md

LQ

LQ is a simple HTTP server that manages named queues of lines of text in memory. By using plain-text request and response bodies, it aims to aid shell scripting scenarios in distributed environments where it's not feasible to set up proper development tools across the nodes (e.g. all you have is curl).

The underlying data structure for each named queue is LinkedHashSet, so the lines in each queue are unique, which may or may not be desirable depending on your requirement. So in essense, LQ is just a map of unbounded LinkedHashSets exposed via HTTP API. No extra effort has been made to make it more efficient or robust.

Usage

Starting the server

# Starts LQ server at port 1234
java -jar lq.jar 1234

Using LQ with curl

# Add lines to the queue named "foobar"
ls -al | curl -XPOST --data-binary @- localhost:1234/foobar

# Look inside the queue
curl localhost:1234/foobar

# Remove the first line in the queue and return it
line="$(curl -XPOST localhost:1234/foobar/shift)"

API Endpoints

Path Method Request Response Description
/ GET Names of queues with their counts List the queues
/ DELETE Number of deleted lines Remove all queues
/:name GET Lines in the queue Return the lines stored in the queue
/:name GET Lines Matched lines in the queue Return the matched lines in the queue (containment test)
/:name PUT Lines Number of lines in the new queue Recreate queue with the lines
/:name POST Lines Number of lines added to the queue Append the lines to the queue
/:name DELETE Number of lines deleted Delete the queue
/:name DELETE Lines Number of lines deleted Delete the lines from the queue
/:name/shift POST Removed line Remove the first line in the queue
/:name1/to/:name2 POST Moved line Move the first line of the first queue to the second queue
/:name1/to/:name2 POST Lines Moved lines Move the lines of the first queue to the second queue

Recipes

Simple task queue

LQ=lq.server.host

# Upload the list of URLs to the queue named "urls"
curl -XPOST --data-binary @urls.txt $LQ/urls

# Process each URL in the queue in order
while [ true ]; do
  url="$(curl -XPOST --silent $LQ/urls/shift)"
  [ -z "$url" ] && break
  echo "Processing $url"
  # ...
done

Polling

while [ true ]; do
  url="$(curl -XPOST --silent $LQ/urls/shift)"
  if [ -z "$url" ]; then
    sleep 5
    continue
  fi

  # ...
done

Task state transition

  1. Take the first line in todo and add it to ongoing
  2. When the task for the line is complete, add it to complete
  3. When todo becomes empty, check if there are incomplete tasks left in ongoing due to unexpected errors.
  4. Move every line in ongoing back to todo
  5. Repeat
LQ=lq.server

# Reset LQ server
curl -XDELETE $LQ/

# Build task queue
cat tasks.txt | curl -XPOST --data-binary @- $LQ/todo

# Process each task
while [ true ]; do
  # Move one task to ongoing
  url="$(curl -XPOST --silent $LQ/todo/to/ongoing)"
  [ -z "$task" ] && break

  # Process the task ...

  # Move the task to done
  curl -XPOST --data-binary "$task" $LQ/ongoing/to/complete
done

# Check the number of completed tasks, are we done?
curl $LQ/complete | wc -l

# Copy lines in ongoing back to todo
curl $LQ/ongoing | curl -XPOST --data-binary @- $LQ/todo

# Delete ongoing
curl -XDELETE $LQ/ongoing

# Repeat until all is done

Development

# Start nREPL + Ring server
make repl

# Build uberjar
make

License

MIT