Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Peas in a #852

wants to merge 35 commits into


None yet
2 participants

temas commented Mar 20, 2013

No description provided.

temas and others added some commits Mar 18, 2013

@temas temas Turn on some debug df16d01
@temas temas Raw log the synclet start/stop 7e27350
@temas temas Log more around pipeline inject 75c7e29
@temas temas Make the dMap not grow stack and tick friendly fc1b1e9
@kristjan kristjan Make friendsPump.vpump asynchronous
This is blocking up the whole pipeline.
@kristjan kristjan Fix missing async callback f26eb84
@kristjan kristjan No seriously, fix async callback d82c048
@temas temas Show the inject length for locals 6b6c904
@temas temas Make sure we have a valid base to length 01170d5
@temas temas Log the exit from an inject d6c6109
@temas temas Basic pump loging for debugging bf2f91c
@temas temas Fix the base in the logging for the pump 0788de8
@temas temas Actually give us a string f6a1e81
@temas temas Fix the logic here bb35e5c
@temas temas Trying to find the stuck bits d946a0a
@temas temas Fix the pump id in the logging bf972e9
@temas temas Short circuit some inject cases 4c1939f
@temas temas Stray code ef27e16
@kristjan kristjan Simplify firstRun check
This construction is silly and angers lint.
@kristjan kristjan Fix missing cbEach in dMap.pump
Didn't catch all our paths. This was causing a hang during the pipeline.
@kristjan kristjan Fix numbering in pipeline logging
1-based counting was confusing and we were talking about the wrong pumps.. Now
the `injector` is 0.
@temas temas isArray... my friend 2df3790
@temas temas Late prep once we have routes and some debug 74b5093
@temas temas More specific ijod timings 1fe8ded
@kristjan kristjan Redo Foursquare checkins synclet
Another recursive monstrosity destroyed.

The strategy here is to start at `now` on the first run and use the earliest
known checkin time with the API's `beforeTimestamp` to page backwards. Meantime,
we keep track on the side of the latest checkin we've seen. When we hit either
the end of history or a checkin we've seen before, we clear the cursor, set the
"we've seen this" time to the latest known checkin, and start over.

Since it looks more and more like large chunks of data hose the pipeline, I
modulated page size based on whether it's the first sync or not. At first sync,
we want to move as fast as we can, so we max out Foursquare's API at 250. Later
on, it's unlikely you've checked in 100 places since we last synced, so we can
drop to the smaller page size and save ourselves effort.
@temas temas Log addData timing more specifically 574de21
@temas temas Synchronize addData abd97c0
@temas temas We don't have an err here 0de062f
@temas temas Missed a callback removal 4fb2400
@temas temas Bit more sync cea681c
@temas temas Fix the instrument for s3 put timing 25ac7f0
@temas temas No ijod events right now c905750
@temas temas Move most of the console.log to logger.debug ee20ecb
@kristjan kristjan Merge branch 'master' into peas-in-a
* master:
  More package.json updates
  Updates for package.json
  add optional tumblr_params for posting
  No more logger.verbose
  Add localLevel, logger.vital
  Improvements to logger
  prefix all logger.warn calls with WARNING.
  Don't crash testSynclet when missing a config
  Be super cautious with the pipeline

@kristjan kristjan Update Foursquare checkins test
New URL's getting hit

temas commented Mar 20, 2013

Disregard, will be split into multiple pull requests.

@temas temas closed this Mar 20, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment