Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streams seem to stop working after a few hours #49

Open
KingScooty opened this issue May 16, 2012 · 9 comments · May be fixed by #109
Open

Streams seem to stop working after a few hours #49

KingScooty opened this issue May 16, 2012 · 9 comments · May be fixed by #109

Comments

@KingScooty
Copy link

I've been testing it locally and on nodejitsu, but after a few hours, the stream seems to stop working. Is this because i'm not handling stream disconnections when a user disconnects by closing their tab/window ?

This is the code i'm using...

    twitter
        .stream('statuses/filter', {track: 'diablo'}, function(stream) {
            stream.on('data', function (data) {
                var encoded = strencode(data);
                var tweet = JSON.parse(encoded);
                var array = { 
                    "id": tweet.id,
                    "screen_name": tweet.user.screen_name,
                    "text" : tweet.text, 
                    "profile_image_url" : tweet.user.profile_image_url 
                };
                io.sockets.send(strencode(array));
            });
            stream.on('end', function (response) {
              // Handle a disconnection
            });
            stream.on('destroy', function (response) {
              // Handle a 'silent' disconnection from Twitter, no end/error event fired
            });
        });
@EdHubbell
Copy link

I was dealing with the same issue a week or 2 ago. I'm running my processes in forever.js, which will restart a node module when it shuts down. So with code like this:

  stream.on 'error', (method, code) ->
    errorTime = new Date()
    console.log 'Error: ' + code + ' ' + method + ' ' + errorTime.getHours() + ':' + errorTime.getMinutes() + ':' + errorTime.getSeconds()
    process.exit 1

  stream.on 'end', (res) ->
    endTime = new Date()
    console.log 'End' + ' ' + endTime.getHours() + ':' + endTime.getMinutes() + ':' + endTime.getSeconds()
    # Handle disconnect
    process.exit 1

  stream.on 'destroy', (res) ->
    destroyTime = new Date()
    console.log 'Destroy' + ' ' + destroyTime.getHours() + ':' + destroyTime.getMinutes() + ':' + destroyTime.getSeconds()
    # Handle 'silent' disconnect from Twitter, no end/error event fired
    process.exit 1

I'm pretty happy.

@mef
Copy link

mef commented Jun 4, 2012

I had the same problem, it is solved by @horixon's immortal-ntwitter: https://github.com/horixon/immortal-ntwitter
my stream is now running well.

@fender
Copy link

fender commented Oct 2, 2012

Is the immortal-ntwitter still needed to fix this? I saw some recent commits to ntwitter claiming reconnects have been improved?

@ram-nadella
Copy link

I have definitely been encountering this issue and it seems to silently stop because I have log statements in my destroy and end event handlers

My code looks like this

try {

    var twitter_client = new ntwitter(twitter_config);

    twitter_client.stream('statuses/filter', {'track': keywords_to_track}, function(stream) {
        stream.on('data', function (data) {
            redis_client.publish("ftweets", JSON.stringify(data));
        });

        stream.on('end', function (response) {
            console.time("Twitter stream end");
        });

        stream.on('destroy', function (response) {
            console.time("Twitter stream destroy");
        });

    });


} catch(ex) {
    console.time(ex);
}

I run this using forever, right after I start I verify it's working by connecting to the Redis server using redis-cli and issuing a MONITOR command, I can see the tweets coming in like crazy and then after a day or two I go back to check if it's still running and there is no activity and also when I do forever list it says the process is still running and there is nothing in the log file. Strange!

@horixon
Copy link

horixon commented Dec 14, 2012

Could it be a stall?

"If 90 seconds pass with no data received, including newlines, disconnect and reconnect immediately according to the backoff strategies in the next section."

I haven't added this feature into my project immortal-ntwitter yet but just opened an issue for it.

@horixon horixon linked a pull request Jun 11, 2013 that will close this issue
@ghost
Copy link

ghost commented Sep 17, 2013

I've been testing this, and those tests seem to confirm that it's indeed a stall. I reset a 90 second interval upon receiving data; the timeout expires and I'm not seeing any new tweets coming in.

@horixon
Copy link

horixon commented Sep 17, 2013

Pull Request #109 does not handle reconnection of the stream. It only destroys the stream and emits an event. Its left up to the ntwitter consumer to recover in a way that is appropriate for there use case. One example of recovery is in immortal-twitter which reconnects the stream when its stalled.

@ram-nadella
Copy link

I ended up writing my own module to deal with the streaming part of the Twitter API (which is all I care about). Same API as ntwitter's .stream()

https://github.com/ram-nadella/tstream

It will automatically reconnect on disconnection, very naively at the moment, still a work in progress.

Emits an event for all the "Disconnect messages (disconnect)" as listed here:
https://dev.twitter.com/docs/streaming-apis/messages#Disconnect_messages_disconnect

What I've found is many instances of the Shutdown case "The feed was shutdown (possibly a machine restart)"

@ghost
Copy link

ghost commented Sep 18, 2013

@horixon , your pull request #109 seems pretty close to the general philosophy of ntwitter of letting the user decide what to do with errors - hope we finally get to see some additional maintainers on this per #100...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants