RESTful polling #158

slef · 2014-02-03T00:20:48Z

Added a RestPollingAgent that (repeatedly) polls a URL and creates an event whenever the server responds.

… the restful polling agent.

cantino · 2014-02-03T03:23:56Z

bin/schedule.rb

+    active_agents = Set.new
+    EM.run do
+        EM.add_periodic_timer(10) {
+            Agent.find_each do |agent|


What about removing Agents that no longer exist?

Ha! Good point :) I'll fix it.

cantino · 2014-02-03T03:25:05Z

Thanks @slef, I left some comments. Looks like a good step forward! Can you add some specs?

cantino · 2014-02-17T00:17:52Z

How's it going?

Added deactivation of destroyed continuous agents in EM. Handling http and parsing errors in RestPollingAgent.

slef · 2014-02-20T10:43:26Z

The update should handle the destruction of agents, and does some error handling.
I'll try to add specs soon.

cantino · 2014-02-20T22:24:26Z

Gemfile

@@ -36,6 +36,7 @@ gem "twitter"
 gem 'twitter-stream', '>=0.1.16'
 gem 'em-http-request'
 gem 'weibo_2'
+gem 'cobravsmongoose'


CobraVsMongoose hans't been updated since 2006. Can you use a different gem, or, ideally, just standard library tools or nokogiri?

Would it be better to use https://github.com/msievers/badgerfish ?
It is more recent but has less features (only does XML -> json conversion) which is ok for our purpose. I think they are both quite short and use standard libraries. What do you think?

Is nokogiri too much of an inconvenience?

On Thu, Feb 20, 2014 at 6:37 PM, slef notifications@github.com wrote:

In Gemfile:

@@ -36,6 +36,7 @@ gem "twitter"
gem 'twitter-stream', '>=0.1.16'
gem 'em-http-request'
gem 'weibo_2'
+gem 'cobravsmongoose'

Would it be better to use https://github.com/msievers/badgerfish ?
It is more recent but has less features (only does XML -> json conversion)
which is ok for our purpose. I think they are both quite short and use
standard libraries. What do you think?

Reply to this email directly or view it on GitHubhttps://github.com//pull/158/files#r9934744
.

slef · 2014-02-21T07:11:14Z

I thought it would be convenient to store the entire XML info in the payload as a json object and then use another agent to reformat that info. The badgerFish convention seems to be the best way of doing that in a lossless manner, see http://wiki.open311.org/index.php?title=JSON_and_XML_Conversion
In particular in my application, I need to access attributes, which are lost in simpler solutions such as Hash.from_xml(s).to_json
If we don't want to use an already existing converter, the alternatives are:

rewrite one from scratch -- should take about a page of code, same as the existing gems
save the raw XML in the payload and create another agent to extract information from there (like formatting agent but for XML)
add some more options to extract the relevant information directly
What do you think is best?

cantino · 2014-02-21T08:01:46Z

Okay, let's use badgerfish.

On Thu, Feb 20, 2014 at 11:11 PM, slef notifications@github.com wrote:

I thought it would be convenient to store the entire XML info in the
payload as a json object and then use another agent to reformat that info.
The badgerFish convention seems to be the best way of doing that in a
lossless manner, see
http://wiki.open311.org/index.php?title=JSON_and_XML_Conversion
In particular in my application, I need to access attributes, which are
lost in simpler solutions such as Hash.from_xml(s).to_json
If we don't want to use an already existing converter, the alternatives
are:

rewrite one from scratch -- should take about a page of code, same
as the existing gems

save the raw XML in the payload and create another agent to extract
information from there (like formatting agent but for XML)

add some more options to extract the relevant information directly
What do you think is best?

Reply to this email directly or view it on GitHubhttps://github.com//pull/158#issuecomment-35704326
.

slef · 2014-02-21T09:26:07Z

Done. Btw it seems like Ox (used by Badgerfish) is faster than nokogiri. http://www.ohler.com/dev/xml_with_ruby/xml_with_ruby.html

cantino · 2014-02-23T22:19:04Z

app/models/agents/rest_polling_agent.rb

+      http.errback {
+          error "Failed: #{http.error}"
+          if !@destroyed
+            pp "errback again "+name


Should this be error?

No. This is debugging code sorry. I wanted to see when it restarts after an error (which might not need to be logged, e.g. in case of a timeout). I'll change it.

cantino · 2014-02-23T22:25:52Z

Thanks @slef. Can you add some specs for your new behavior?

slef · 2014-03-06T06:49:03Z

I'm having some problems figuring out how to write specs for the agent running inside the EM loop. A long polling server could be simulated using webmock and a Queue or an EM:Queue:

@jsonqueue = Queue.new 
stub_request(:any, 'www.jsonexample.net').to_return { |request| @jsonqueue.pop }

Then the next http request should hold until I push something on the queue.
If I do this, and em-start an agent, how do I check that the event was created?

cantino · 2014-03-07T03:08:25Z

Yea, that's tricky. I'd probably extract the interesting parts into methods on a class, test those, and then either not test, or stub, the EM stuff.

0xdevalias · 2014-06-01T06:35:01Z

@slef Did you ever figure out a nice way to write specs for this?

slef · 2014-06-02T01:17:24Z

Not really... I'll try to get back to this within a couple of weeks. Any ideas?

0xdevalias · 2014-06-02T05:12:04Z

Unfortunately not off hand. Specs are my downfall as well :P

cantino · 2014-06-02T05:13:58Z

@dsander or I can probably put something together. Is this way of doing EM Agents how we want to proceed? Should all agents head in the EM direction and phase out DJ?

slef · 2014-06-03T12:30:22Z

I'm not sure but it looks like EM is a bit less flexible/accurate when it comes to scheduling tasks at specific times. But for all agents that generate events from other events, webhooks, streams, etc. (basically everything but cron jobs), I don't see why they shouldn't all be continuous!.
And if we can find a way to make scheduling accurate in EM, then we could put everything in there. Right?

dsander · 2014-06-03T17:02:34Z

Like I state in #52 I am not the biggest fan of EM (limited to EM enabled libraries, no additional concurrency possible) but it is much easier to implement then using threads so I think it is ok to start with it.
I however still believe the EM loop should be run in a separate process (or thread with the threaded workers).

Jngai · 2016-04-06T21:02:32Z

@slef are you still working on this? looks like its almost done minus the spec.

slef · 2016-04-19T09:14:38Z

I have taken a break for a while. Feel free to take a shot at it if you like.

Added EventMachine to scheduler, the continuous! option to agents and…

6042463

… the restful polling agent.

cantino reviewed Feb 3, 2014
View reviewed changes

Robustifying

7c70272

Added deactivation of destroyed continuous agents in EM. Handling http and parsing errors in RestPollingAgent.

cantino reviewed Feb 20, 2014
View reviewed changes

Replaced cobravsmongoose by badgerfish

d06062a

cantino reviewed Feb 23, 2014
View reviewed changes

0xdevalias added agent labels Jun 1, 2014

slef mentioned this pull request Jun 2, 2014

RESTful polling #52

Open

cantino mentioned this pull request Aug 21, 2014

Dropbox Agent #321

Open

knu force-pushed the master branch from bd2a35d to cdc06a6 Compare April 28, 2024 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RESTful polling #158

RESTful polling #158

slef commented Feb 3, 2014

cantino Feb 3, 2014

slef Feb 3, 2014

cantino commented Feb 3, 2014

cantino commented Feb 17, 2014

slef commented Feb 20, 2014

cantino Feb 20, 2014

slef Feb 21, 2014

cantino Feb 21, 2014

slef commented Feb 21, 2014

cantino commented Feb 21, 2014

slef commented Feb 21, 2014

cantino Feb 23, 2014

slef Feb 24, 2014

cantino commented Feb 23, 2014

slef commented Mar 6, 2014

cantino commented Mar 7, 2014

0xdevalias commented Jun 1, 2014

slef commented Jun 2, 2014

0xdevalias commented Jun 2, 2014

cantino commented Jun 2, 2014

slef commented Jun 3, 2014

dsander commented Jun 3, 2014

Jngai commented Apr 6, 2016

slef commented Apr 19, 2016

RESTful polling #158

Are you sure you want to change the base?

RESTful polling #158

Conversation

slef commented Feb 3, 2014

cantino Feb 3, 2014

Choose a reason for hiding this comment

slef Feb 3, 2014

Choose a reason for hiding this comment

cantino commented Feb 3, 2014

cantino commented Feb 17, 2014

slef commented Feb 20, 2014

cantino Feb 20, 2014

Choose a reason for hiding this comment

slef Feb 21, 2014

Choose a reason for hiding this comment

cantino Feb 21, 2014

Choose a reason for hiding this comment

slef commented Feb 21, 2014

cantino commented Feb 21, 2014

slef commented Feb 21, 2014

cantino Feb 23, 2014

Choose a reason for hiding this comment

slef Feb 24, 2014

Choose a reason for hiding this comment

cantino commented Feb 23, 2014

slef commented Mar 6, 2014

cantino commented Mar 7, 2014

0xdevalias commented Jun 1, 2014

slef commented Jun 2, 2014

0xdevalias commented Jun 2, 2014

cantino commented Jun 2, 2014

slef commented Jun 3, 2014

dsander commented Jun 3, 2014

Jngai commented Apr 6, 2016

slef commented Apr 19, 2016