Browse files

more readme

  • Loading branch information...
JustinBeckwith committed Jan 29, 2013
1 parent 165c6e1 commit 308627a037c4a9d4b4fd4e6b3521b395275415da
Showing with 14 additions and 9 deletions.
  1. +14 −9
@@ -4,8 +4,6 @@
[2]: (View the Demo)
[Wazstagram]( is a fun experiment with node.js on [Windows Azure]( and the [Instagram Realtime API]( The project uses various services in Windows Azure to create a scalable window into Instagram traffic across multiple cities.
@@ -56,7 +54,8 @@ After you have the azure module, you're ready to rock.
The [backend]( part of this project is a worker role that accepts HTTP Post messages from the Instagram API. The idea is that their API batches messages, and sends them to an endpoint you define. Here's [some details]( on how their API works. I chose to use [express]( to build out the backend routes, because it's convenient. There are a few pieces to the backend that are interesting:
-1. Use [nconf]( to store secrets. Look at the .gitignore.
+1. ##### Use [nconf]( to store secrets. Look at the .gitignore.
If you're going to build a site like this, you are going to need to store a few secrets. The backend includes things like the Instagram API key, my Windows Azure Storage account key, and my Service Bus keys. I create a keys.json file to store this, though you could add it to the environment. I include an example of this file with the project. **DO NOT CHECK THIS FILE INTO GITHUB!** Seriously, [don't do that]( Also, pay **close attention** to my [.gitignore file]( You don't want to check in any *.cspkg or *.csx files, as they contain archived versions of your site that are generated while running the emulator and deploying. Those archives contain your keys.json file. That having been said - nconf does makes it really easy to read stuff from your config:
@@ -68,7 +67,8 @@ The [backend](
var stKey = nconf.get('AZURE_STORAGE_KEY');
-2. Use [winston]( and [winston-skywriter]( for logging
+2. ##### Use [winston]( and [winston-skywriter]( for logging.
The cloud presents some challenges at times. Like *how do I get console output* when something goes wrong. Every node.js project I start these days, I just use winston from the get go. It's awesome because it lets you pick where your console output and logging gets stored. I like to just pipe the output to console at dev time, and write to [Table Storage]( in production. Here's how you set it up:
@@ -87,7 +87,8 @@ The [backend]('Started wazstagram backend');
-3. Use [Service Bus]( - it's pub/sub (+) a basket of kittens
+3. ##### Use [Service Bus]( - it's pub/sub (+) a basket of kittens.
[Service Bus]( is Windows Azure's swiss army knife of messaging. I usually use it in the places where I would otherwise use the PubSub features of Redis. It does all kinds of neat things like [PubSub](, [Durable Queues](, and more recently [Notification Hubs]( I use the topic subscription model to create a single channel for messages. Each worker node publishes messages to a single topic. Each web node creates a subscription to that topic, and polls for messages. There's great [support for Service Bus]( in the [Windows Azure Node.js SDK](
To get the basic implementation set up, just follow the [Service Bus Node.js guide]( The interesting part of my use of Service Bus is the subscription clean up. Each new front end node that connects to the topic creates it's own subscription. As we scale out and add a new front end node, it creates another subscription. This is a durable object in Service bus that hangs around after the connection from one end goes away (this is a feature). To make sure sure you don't leave random subscriptions lying around, you need to do a little cleanup:
@@ -118,7 +119,8 @@ The [backend](
-4. The [NewImage endpoint](
+4. ##### The [NewImage endpoint](
All of the stuff above is great, but it doesn't cover what happens when the Instagram API actually hits our endpoint. The route that accepts this request gets metadata for each image, and pushes it through the Service Bus topic:
@@ -134,7 +136,8 @@ The [backend](
## The Frontend
The [frontend]( part of this project is (despite my 'web node' reference) a worker role that accepts accepts the incoming traffic from end users on the site. I chose to use worker roles because I wanted to take advantage of Web Sockets. At the moment, Cloud Services Web Roles do not provide that functionality. I could stand up a VM with Windows Server 8 and IIS 8, but see my aformentioned anxiety about managing my own VMs. The worker roles use []( and [express]( to provide the web site experience. The front end uses the same NPM modules as the backend: [express](, [winston](, [winston-skywriter](, [nconf](, and [azure]( In addition to that, it uses []( and [ejs]( to handle the client stuff. There are a few pieces to the frontend that are interesting:
-1. Setting up
+1. ##### Setting up
+ provides the web socket (or xhr) interface that we're going to use to stream images to the client. When a user initially visits the page, they are going to send a `setCity` call, that lets us know the city to which they want to subscribe (by default all [cities in the system]( are returned). From there, the user will be sent an initial blast of images that are cached on the server. Otherwise, you wouldn't see images right away:
@@ -152,7 +155,8 @@ The [frontend](
-2. Creating a Service Bus Subscription
+2. ##### Creating a Service Bus Subscription
To receive messages from the worker nodes, we need to create a single subscription for each front end node process. This is going to create subscription, and start listening for messages:
@@ -167,7 +171,8 @@ The [frontend](
-3. Moving data between Service Bus and Socket.IO
+3. ##### Moving data between Service Bus and Socket.IO
As data comes in through the service bus subscription, you need to pipe it up to the appropriate connected clients. Pay special attention to `` - when the user joined the page, they selected a city. This call grabs all users subscribed to that city. The other **important thing to notice** here is the way `getFromTheBus` calls itself in a loop. There's currently no way to say "just raise an event when there's data" with the Service Bus Node.js implementation, so you need to use this model.

0 comments on commit 308627a

Please sign in to comment.