Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
A pure stream http push technology for your Nginx setup. Comet made easy and really scalable.
C Ruby JavaScript HTML Other

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
include
misc
src
test
.gitignore
AUTHORS
CHANGELOG.textile
COPYING
LICENSE
README.textile
build.sh
config
pack.sh

README.textile

Nginx Push Stream Module

A pure stream http push technology for your Nginx setup.

Comet made easy and really scalable.

Changelog

Always take a look at CHANGELOG.textile.

Installing


    # clone
    git clone http://github.com/wandenberg/nginx-push-stream-module.git
    cd nginx-push-stream-module

    # build with 1.0.x, 0.9.x, 0.8.x series or any other nginx version later than 0.7.67
    ./build.sh master 1.0.0
    cd build/nginx-1.0.0

    # finish
    sudo make install

    # check
    sudo /usr/local/nginx/sbin/nginx -v
        nginx version: nginx/1.0.0
    sudo /usr/local/nginx/sbin/nginx -c nginx-push-stream-module/misc/nginx.conf -t
        the configuration file nginx-push-stream-module/misc/nginx.conf syntax is ok
        configuration file nginx-push-stream-module/misc/nginx.conf test is successful

    # run
    sudo /usr/local/nginx/sbin/nginx -c nginx-push-stream-module/misc/nginx.conf

Basic Configuration


    location /channels-stats {
        # activate channels statistics mode for this location
        push_stream_channels_statistics;

        # query string based channel id
        set $push_stream_channel_id             $arg_id;
    }

    location /pub {
        # activate publisher mode for this location
        push_stream_publisher;

        # query string based channel id
        set $push_stream_channel_id             $arg_id;
        push_stream_max_channel_id_length       200;
        # message template
        push_stream_message_template            "<script>p(~id~,'~channel~','~text~');</script>";
        # store messages in memory
        push_stream_store_messages              on;
        # max messages to store in memory
        push_stream_max_message_buffer_length   20;
        # message ttl
        push_stream_min_message_buffer_timeout  5m;

        # Message size limit
        # client_max_body_size MUST be equal to client_body_buffer_size or
        # you will be sorry.
        client_max_body_size                    32k;
        client_body_buffer_size                 32k;
    }

    location ~ /sub/(.*) {
        # activate subscriber mode for this location
        push_stream_subscriber;

        # positional channel path
        set $push_stream_channels_path              $1;
        push_stream_max_channel_id_length           200;
        # header to be sent when receiving new subscriber connection
        push_stream_header_template                 "<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\r\n<meta http-equiv=\"Cache-Control\" content=\"no-store\">\r\n<meta http-equiv=\"Cache-Control\" content=\"no-cache\">\r\n<meta http-equiv=\"Pragma\" content=\"no-cache\">\r\n<meta http-equiv=\"Expires\" content=\"Thu, 1 Jan 1970 00:00:00 GMT\">\r\n<script type=\"text/javascript\">\r\nwindow.onError = null;\r\ndocument.domain = 'localhost';\r\nparent.PushStream.register(this);\r\n</script>\r\n</head>\r\n<body onload=\"try { parent.PushStream.reset(this) } catch (e) {}\">";
        # message template
        push_stream_message_template                "<script>p(~id~,'~channel~','~text~');</script>";
        # content-type
        push_stream_content_type                    "text/html; charset=utf-8";
        # subscriber may create channels on demand or only authorized
        # (publisher) may do it?
        push_stream_authorized_channels_only        off;
        # ping frequency
        push_stream_ping_message_interval           10s;
        # connection ttl to enable recycle
        push_stream_subscriber_connection_timeout   15m;
        # broadcast
        push_stream_broadcast_channel_prefix        "broad_";
        push_stream_broadcast_channel_max_qtd       3;

        # solving some leakage problem with persistent connections in
        # Nginx's chunked filter (ngx_http_chunked_filter_module.c)
        chunked_transfer_encoding                   off;
    }

Basic Usage

You can feel the flavor right now at the command line. Try using more than
one terminal and start playing http pubsub:


    # Pub
    curl -s -v -X POST "http://localhost/pub?id=my_channel_1" -d "Hello World!"

    # Sub
    curl -s -v "http://localhost/sub/my_channel_1.b20"

    # Channel Stats for publisher (json format)
    curl -s -v "http://localhost/pub?id=my_channel_1"

    # All Channels Stats summarized (json format)
    curl -s -v "http://localhost/channels-stats"

    # All Channels Stats detailed (json format)
    curl -s -v "http://localhost/channels-stats?id=ALL"

    # Channels Stats (json format)
    curl -s -v "http://localhost/channels-stats?id=my_channel_1"

Configuration

Variables

variable values location
push_stream_channel_id channel id push_stream_publisher, push_stream_channels_statistics
push_stream_channels_path set of channels id and backtrack desired messages push_stream_subscriber

push_stream_channel_id

A string to uniquely identify a communication channel. Must be present on location of the push_stream_publisher and push_stream_channels_statistics.


set $push_channel_id $arg_id;
#channel id is now the url query string parameter "id"
#(/pub?id=channel_id_string or /channels-stats?id=channel_id_string)

push_stream_channels_path

A string representing a set of channels id and backtrack desired messages separated by slash, example /channel1.b3/channel2.b5/channel3.b2.
The backtrack means the amount of old messages from each of the channels that will be delivered to the subscriber. On the example will be 3 messages from channel1, 5 from channel2 and 2 from channel3.
Backtrack isn’t needed, you can only sign channels without get old messages, or you can mix things.
More accepted examples: /channel1, /channel1/channel2, /channel1.b5/channel2, /channel1/channel2.b6_, …
Must be present on location of the push_stream
subscriber.


location /sub/(.*) {
  set $push_stream_channels_path $1;
}
#channels path is now part of url
#(/sub/channel_id_string or /sub/channel_id_string.b2/other_channel)

Directives

directive default value values context location
push_stream_channels_statistics - - location -
push_stream_publisher - - location -
push_stream_subscriber - - location -
push_stream_max_reserved_memory 32M size greater than 8 * ngx_pagesize http main nginx configuration
push_stream_store_messages off on, off location push_stream_publisher
push_stream_min_message_buffer_timeout unset time constant location push_stream_publisher
push_stream_max_message_buffer_length unset number location push_stream_publisher
push_stream_authorized_channels_only off on, off location push_stream_subscriber
push_stream_header_template unset any string location push_stream_subscriber
push_stream_content_type text/plain any valid content type location push_stream_subscriber
push_stream_ping_message_interval unset time constant location push_stream_subscriber
push_stream_subscriber_connection_timeout unset time constant location push_stream_subscriber
push_stream_broadcast_channel_max_qtd unset number location push_stream_subscriber
push_stream_max_channel_id_length unset number http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration
push_stream_broadcast_channel_prefix unset any string http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration
push_stream_message_template unset any string http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration
push_stream_max_number_of_channels unset number http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration
push_stream_max_number_of_broadcast_channels unset number http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration
push_stream_memory_cleanup_timeout 30 seconds time constant http, location (push_stream_subscriber and push_stream_publisher) or main nginx configuration

push_stream_channels_statistics

New in version 0.2.0
context: location

Defines a location as a source of statistics. You can use this location to get statistics about a specific channel, or about all channels, in a resumed ou summarized way.
To get statistics about all channels in a summarized way you have to make a GET in this location without specify a name in the push_stream_channel_id variable.
To get statistics about all channels in a detailed way you have to specify “ALL” in the push_stream_channel_id.
To get statistics about a channel you have to specify the name in the push_stream_channel_id.

You can get statistics in the formats plain, xml, yaml and json. The default is json, to change this behavior you can use Accept header parameter passing values like “text/plain”, “application/xml”, “application/yaml” and “application/json” respectivelly.


  location /channels-stats {
      push_stream_channels_statistics;
      set $push_stream_channel_id             $arg_id;
  }

  # /channels-stats -> get statistics about all channels in a summarized way
  # /channels-stats?id=ALL -> get statistics about all channels in a detailed way
  # /channels-stats?id=channel_id -> get statistics about a channel

push_stream_publisher

context: location

Defines a location as a message publisher. Requests to a publisher location are treated as messages to be sent to subscribers.
This location supports two http methods:
GET, make possible to get statistics about the channel
POST, publish a message to the channel


  location /pub {
      push_stream_publisher;
      set $push_stream_channel_id             $arg_id;
  }

  # GET  /pub?id=channel_id -> get statistics about a channel
  # POST /pub?id=channel_id -> publish a message to the channel

push_stream_subscriber

context: location

Defines a location as a subscriber. This location represents a subscriber’s interface to a channel’s message queue.
This location only supports GET http method to receive published messages in a stream.

Functionality

push_stream_header_template [ string ]

default: -
context: location
location: push_stream_subscriber

The text that will be sended to subscribers when they arrive.

push_stream_message_template [ string ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The text template that will be used to format the message before be sended to subscribers. The template can contain any number of the reserved words: ~id~, ~text~ and ~channel~, example: "<script>p(~id~,'~channel~','~text~');</script>"

push_stream_content_type [ string ]

default: text/plain
context: location
location: push_stream_subscriber

The content type used on responses to subscribers. Must be complient with push_stream_header_template and push_stream_message_template.

push_stream_ping_message_interval [ time ]

default: -
context: location
location: push_stream_subscriber

The time interval in which a keepalive message is sent to subscribers.

Message/Channel Storage

push_stream_max_reserved_memory [ size ]

default: 32M
context: http

The size of the memory chunk this module will use to store published messages, channels and other shared structures.
When this memory is full any new request for publish a message or subscribe a channel will receive an 500 Internal Server Error response.

push_stream_store_messages [ on | off ]

default: off
context: location
location: push_stream_publisher

Whether or not message queuing is enabled.
If store messages is “On” is needed to set at least one of these two directives push_stream_min_message_buffer_timeout or push_stream_max_message_buffer_length.

push_stream_min_message_buffer_timeout [ time ]

default: -
context: location
location: push_stream_publisher

The length of time a message may be queued before it is considered expired. If you do not want messages to expire, just not set this directive.

push_stream_max_message_buffer_length [ number ]

default: -
context: location
location: push_stream_publisher

The maximum number of messages to store per channel. A channel’s message buffer will retain at most this many most recent messages. If you do not want messages to be discarded by length, just not set this directive.

Security

push_stream_authorized_channels_only [ on | off ]

default: off
context: location
location: push_stream_subscriber

Whether or not a subscriber may create a channel by making a request to a push_stream_subscriber location. If set to on, a publisher must send a POST request before a subscriber can request messages on the channel. Otherwise, all subscriber requests to nonexistent channels will get a 403 Forbidden response.
This restriction is not applied to broadcast channels, but to subscribe to a broadcast channel is necessary to subscribe at least to one normal channel, and if this directive is set to on this channel has to be created before.

push_stream_subscriber_connection_timeout [ time ]

default: -
context: location
location: push_stream_subscriber

The length of time a subscriber will stay connected before it is considered expired and disconnected. If you do not want subscribers to be automatically disconnected, just not set this directive.
But, this operation is very important to help Nginx recycle memory consumed to send messages to susbscriber, allocated at pool request.

push_stream_max_channel_id_length [ number ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

Maximum permissible channel id length (number of characters). Longer ids will receive an 400 Bad Request response.

push_stream_broadcast_channel_prefix [ string ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The string prefix used to identify when a channel is as normal or broadcast channel, example: when you set this directive as “bd_”, “bd_ch1” will be a broadcast channel

push_stream_broadcast_channel_max_qtd [ number ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The maximum number of broadcast channels that a subscriber may sign on the request.
This directive works in conjunction with push_stream_authorized_channels_only to preserve the server from a kind of attack where a subscriber sign one normal channel and many nonexistent broadcast channels.

push_stream_max_number_of_channels [ number ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The maximum number of concurrent channels on the server. If you do not want to limit the number of channels, just not set this directive.

push_stream_max_number_of_broadcast_channels [ number ]

default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The maximum number of concurrent broadcats channels on the server. If you do not want to limit the number of broadcast channels, just not set this directive.

push_stream_memory_cleanup_timeout [ time ]

default: 30 seconds
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration

The length of time a message or a channel will stay on garbage collection area before it is completly discarded, freeing the shared memory. The minimum length is 30 seconds to ensure that no one is using these elements.
This operation is very important to help Nginx recycle memory consumed to create messages and channels, so do not use a large time.

Attention

This module controls everything needed to send the messages to subscribers.
So is strongly recommended to you disable Nginx’s chuncked filter to reduce memory consumption.
You can do this using the chunked_transfer_encoding directive on subscriber location


# solving some leakage problem with persistent connections in
# Nginx's chunked filter (ngx_http_chunked_filter_module.c)
chunked_transfer_encoding                   off;

Tests

The tests for this module are written in Ruby, and are acceptance tests.
To run them is needed to have an environment with:

  • Basic requirements
    • ruby >= 1.8.7
    • rubygems >= 1.6.2
    • rake >= 0.8.7
  • Required gems
    • POpen4 >= 0.1.4
    • em-http-request >= 0.2.14
    • json >= 1.4.3
    • ruby-debug >= 0.10.4

You can install these gems with bundler (bundler is required to be installed before, gem install bundler)


cd test/
bundle install --without docs

or individually


gem install POpen4 -v 0.1.4
gem install em-http-request -v 0.2.14
gem install json -v 1.4.3
gem install ruby-debug -v 0.10.4

Then issue rake tests.
This command run the tests using nginx executable located at /usr/local/nginx/sbin/nginx with 1 worker responding at host localhost and port 9990.
To change this behavior use the commands bellow


rake tests executable="../build/nginx-1.0.0/objs/nginx"   # to change default path for nginx executable
rake tests host=localhost                                 # to change default hostname
rake tests port=9889                                      # to change default port
rake tests workers=2                                      # to change dafault number of workers used

and can combine any of these parameters, like:

rake tests port=9889 executable="../build/nginx-1.0.0/objs/nginx"

Discussion

Nginx Push Stream Module Discussion Group

Contributors

People

Donate

Feel free to give something back ;)
Donate

Something went wrong with that request. Please try again.