New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite bee-queue 1.0.0 #64

Merged
merged 84 commits into from Aug 9, 2017

Conversation

Projects
None yet
6 participants
@skeggse
Member

skeggse commented Jul 13, 2017

Note that we haven't re-attained 100% code coverage, and that's the biggest blocker to considering this ready to merge.

@skeggse skeggse requested a review from LewisJEllis Jul 13, 2017

@@ -1,14 +1,29 @@
0.3.0 / 09-01-2015
1.0.0 / 2017-06-30

This comment has been minimized.

@bradvogel

bradvogel Jul 13, 2017

Contributor

these new features need to be documented in the readme

@bradvogel

bradvogel Jul 13, 2017

Contributor

these new features need to be documented in the readme

Show outdated Hide outdated package.json Outdated
Show outdated Hide outdated lib/job.js Outdated
"description": "A simple, fast, robust job/task queue, backed by Redis.",
"main": "index.js",
"dependencies": {
"redis": "1.0.0"
"promise-callbacks": "^3.0.0",

This comment has been minimized.

@LewisJEllis

LewisJEllis Jul 13, 2017

Member

I'd much prefer if we pin these. I'm guessing it's just oversight, but I'm happy to discuss if you'd explicitly rather not.

@LewisJEllis

LewisJEllis Jul 13, 2017

Member

I'd much prefer if we pin these. I'm guessing it's just oversight, but I'm happy to discuss if you'd explicitly rather not.

This comment has been minimized.

@skeggse

skeggse Jul 13, 2017

Member

We prefer to not pin dependencies, because it seems more the job of the application to ensure that its dependencies are locked. In our case, we use yarn, but other folks may use npm shrinkwrap or lockfiles. For us, not pinning dependencies is beneficial because yarn deduplicates common dependencies.

@skeggse

skeggse Jul 13, 2017

Member

We prefer to not pin dependencies, because it seems more the job of the application to ensure that its dependencies are locked. In our case, we use yarn, but other folks may use npm shrinkwrap or lockfiles. For us, not pinning dependencies is beneficial because yarn deduplicates common dependencies.

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Jul 13, 2017

Member

My 10-minute skim impression/thoughts:

  • Looks great overall, you've kept along a lot of the original ideas/tenets I had when starting this and polished it to be 2017 JS instead of 2015 JS. Most of the spots where the 2-years-later version of me sees room for stuff to be better, it is, especially around tooling/CI/etc.
  • Good move on the promise transition. I skipped them previously to dodge bluebird/a compilation step/node 0.12's sketchy promise situation, but now that they're a standard language feature (along with async/await) I'm happy to see/use them. Curious about your thoughts on the option of moving to an entirely promise-first API and maybe obviating the need for promise-callback?
  • I'm gonna have to go through in more detail to understand the new timer stuff and the new behavior for stalling/delayed jobs; definitely adds complexity, but it looks well-thought-out.

It sounds like you're still working on tests, which do make review easier, so I'm inclined to wait on those, but just let me know whenever you think they're in good shape for review. Maybe that's even now, I'm not sure - happy to review now during WIP or wait, just let me know either way. Just hoping to avoid leaving comments of "there should be a test for this" when you already know :)

Member

LewisJEllis commented Jul 13, 2017

My 10-minute skim impression/thoughts:

  • Looks great overall, you've kept along a lot of the original ideas/tenets I had when starting this and polished it to be 2017 JS instead of 2015 JS. Most of the spots where the 2-years-later version of me sees room for stuff to be better, it is, especially around tooling/CI/etc.
  • Good move on the promise transition. I skipped them previously to dodge bluebird/a compilation step/node 0.12's sketchy promise situation, but now that they're a standard language feature (along with async/await) I'm happy to see/use them. Curious about your thoughts on the option of moving to an entirely promise-first API and maybe obviating the need for promise-callback?
  • I'm gonna have to go through in more detail to understand the new timer stuff and the new behavior for stalling/delayed jobs; definitely adds complexity, but it looks well-thought-out.

It sounds like you're still working on tests, which do make review easier, so I'm inclined to wait on those, but just let me know whenever you think they're in good shape for review. Maybe that's even now, I'm not sure - happy to review now during WIP or wait, just let me know either way. Just hoping to avoid leaving comments of "there should be a test for this" when you already know :)

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Jul 13, 2017

Member

Oh, also definitely looking for more readme updating, especially if you want to go waving a flag of "hey we're updated and back in action". I'd guess my initial release time breakdown was 1:1:1 impl/tests/readme, and I think that was a significant factor to why this initially caught some adoption (well, that and the whim of the editor of the Node Weekly newsletter...).

Member

LewisJEllis commented Jul 13, 2017

Oh, also definitely looking for more readme updating, especially if you want to go waving a flag of "hey we're updated and back in action". I'd guess my initial release time breakdown was 1:1:1 impl/tests/readme, and I think that was a significant factor to why this initially caught some adoption (well, that and the whim of the editor of the Node Weekly newsletter...).

@bradvogel

This comment has been minimized.

Show comment
Hide comment
@bradvogel

bradvogel Jul 17, 2017

Contributor

@LewisJEllis fyi: we're still debugging one last issue with this PR (running this in production!). So wait on reviewing.

Contributor

bradvogel commented Jul 17, 2017

@LewisJEllis fyi: we're still debugging one last issue with this PR (running this in production!). So wait on reviewing.

@skeggse

This comment has been minimized.

Show comment
Hide comment
@skeggse

skeggse Jul 19, 2017

Member

@LewisJEllis it looks like maybe Coveralls isn't functioning now that bee-queue moved to its own organization. I'm not able to get Coveralls to discover bee-queue via my account, can you take a look?

Member

skeggse commented Jul 19, 2017

@LewisJEllis it looks like maybe Coveralls isn't functioning now that bee-queue moved to its own organization. I'm not able to get Coveralls to discover bee-queue via my account, can you take a look?

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Jul 19, 2017

Member

Sure thing - I went into my coveralls account and hit some sort of button that turned https://coveralls.io/github/lewisjellis/bee-queue into https://coveralls.io/github/bee-queue/bee-queue and synced/updated everything; I think it should try to do a report on the next push, so give it a try. If that doesn't work, you also might try moving to a new PR that's on a branch in this repo rather than on the mixmaxhq fork.

Member

LewisJEllis commented Jul 19, 2017

Sure thing - I went into my coveralls account and hit some sort of button that turned https://coveralls.io/github/lewisjellis/bee-queue into https://coveralls.io/github/bee-queue/bee-queue and synced/updated everything; I think it should try to do a report on the next push, so give it a try. If that doesn't work, you also might try moving to a new PR that's on a branch in this repo rather than on the mixmaxhq fork.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 19, 2017

Coverage Status

Changes Unknown when pulling 92535f4 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Jul 19, 2017

Coverage Status

Changes Unknown when pulling 92535f4 on mixmaxhq:master into ** on bee-queue:master**.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 20, 2017

Coverage Status

Changes Unknown when pulling 0db0f76 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Jul 20, 2017

Coverage Status

Changes Unknown when pulling 0db0f76 on mixmaxhq:master into ** on bee-queue:master**.

@skeggse

This comment has been minimized.

Show comment
Hide comment
@skeggse

skeggse Jul 20, 2017

Member

We're running into what may be a performance regression Node-side (Redis is still performing quite nicely) - I suspect that the switch to promises, especially ES6 promises, may be creating substantially more objects than we expected. I'm seeing GC overhead of up to 15% in production. Thoughts about performance and promises, @LewisJEllis? It looks like maybe bluebird is more optimized relative to es6 promises, so maybe we could support overriding the Promise implementation at runtime.

Member

skeggse commented Jul 20, 2017

We're running into what may be a performance regression Node-side (Redis is still performing quite nicely) - I suspect that the switch to promises, especially ES6 promises, may be creating substantially more objects than we expected. I'm seeing GC overhead of up to 15% in production. Thoughts about performance and promises, @LewisJEllis? It looks like maybe bluebird is more optimized relative to es6 promises, so maybe we could support overriding the Promise implementation at runtime.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jul 20, 2017

Coverage Status

Changes Unknown when pulling fad6899 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Jul 20, 2017

Coverage Status

Changes Unknown when pulling fad6899 on mixmaxhq:master into ** on bee-queue:master**.

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Jul 20, 2017

Member

That's actually entirely possible. I recall playing with Bluebird's benchmarks during initial dev here and deciding that the speed advantage of raw callbacks could be significant in extremely high-throughput scenarios, but it was a narrow edge; I kind of assumed serialization and IO would still dominate and so I focused most of my efforts on minimizing roundtrips and maximizing pipelining & effective use of Lua scripting.

It sounds like that might not be the case, though, for native promises; I just ran some of bluebird's benchmarks:

results for 50000 parallel executions, 1 ms per I/O op

file                                       time(ms)  memory(MB)
callbacks-baseline.js                          1110      115.84
promises-bluebird-generator.js                 1940      209.61
promises-bluebird.js                           2332      220.53
promises-ecmascript6-native.js                 6899      430.85

Platform info:
Darwin 16.6.0 x64
Node.JS 8.1.2
V8 5.8.283.41
Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz × 4

and at 50 parallel:

results for 10000 parallel executions, 1 ms per I/O op

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          776      136.92
promises-bluebird.js                           999      181.27
promises-bluebird-generator.js                1012      187.42
promises-ecmascript6-native.js                5035      574.09

Platform info:
Darwin 16.6.0 x64
Node.JS 8.1.2
V8 5.8.283.41
Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz × 4

It looks like native promises are 5-6x slower than raw callbacks and 3-5x slower than bluebird here. I don't think the delta between callbacks and bluebird promises would be significant, but maybe with native promises it would be. Definitely worth benchmarking, and ultimately I think using bluebird could be reasonable for performance reasons. If you're majorly concerned about memory usage then I guess sticking to raw callbacks could have a bigger advantage, but I don't think memory of the node process is a big concern relative to throughput and Redis load.

Member

LewisJEllis commented Jul 20, 2017

That's actually entirely possible. I recall playing with Bluebird's benchmarks during initial dev here and deciding that the speed advantage of raw callbacks could be significant in extremely high-throughput scenarios, but it was a narrow edge; I kind of assumed serialization and IO would still dominate and so I focused most of my efforts on minimizing roundtrips and maximizing pipelining & effective use of Lua scripting.

It sounds like that might not be the case, though, for native promises; I just ran some of bluebird's benchmarks:

results for 50000 parallel executions, 1 ms per I/O op

file                                       time(ms)  memory(MB)
callbacks-baseline.js                          1110      115.84
promises-bluebird-generator.js                 1940      209.61
promises-bluebird.js                           2332      220.53
promises-ecmascript6-native.js                 6899      430.85

Platform info:
Darwin 16.6.0 x64
Node.JS 8.1.2
V8 5.8.283.41
Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz × 4

and at 50 parallel:

results for 10000 parallel executions, 1 ms per I/O op

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          776      136.92
promises-bluebird.js                           999      181.27
promises-bluebird-generator.js                1012      187.42
promises-ecmascript6-native.js                5035      574.09

Platform info:
Darwin 16.6.0 x64
Node.JS 8.1.2
V8 5.8.283.41
Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz × 4

It looks like native promises are 5-6x slower than raw callbacks and 3-5x slower than bluebird here. I don't think the delta between callbacks and bluebird promises would be significant, but maybe with native promises it would be. Definitely worth benchmarking, and ultimately I think using bluebird could be reasonable for performance reasons. If you're majorly concerned about memory usage then I guess sticking to raw callbacks could have a bigger advantage, but I don't think memory of the node process is a big concern relative to throughput and Redis load.

@skeggse

This comment has been minimized.

Show comment
Hide comment
@skeggse

skeggse Jul 21, 2017

Member

I ran a battery of benchmarks against different node versions, and three variations of bee queue (alongside bull and kue) - I'll report back with my findings once I finish digging through them.

Member

skeggse commented Jul 21, 2017

I ran a battery of benchmarks against different node versions, and three variations of bee queue (alongside bull and kue) - I'll report back with my findings once I finish digging through them.

skeggse added some commits Jun 28, 2017

skeggse added some commits Aug 8, 2017

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 4574ac0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 4574ac0 on mixmaxhq:master into ** on bee-queue:master**.

Show outdated Hide outdated lib/queue.js Outdated
Show outdated Hide outdated lib/queue.js Outdated

skeggse added some commits Aug 8, 2017

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

4 similar comments
@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 66d8cf0 on mixmaxhq:master into ** on bee-queue:master**.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Aug 8, 2017

Coverage Status

Changes Unknown when pulling 936b9b6 on mixmaxhq:master into ** on bee-queue:master**.

coveralls commented Aug 8, 2017

Coverage Status

Changes Unknown when pulling 936b9b6 on mixmaxhq:master into ** on bee-queue:master**.

@LewisJEllis

Looks ready!

@skeggse skeggse merged commit ffe5194 into bee-queue:master Aug 9, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@skeggse

This comment has been minimized.

Show comment
Hide comment
@skeggse

skeggse Aug 9, 2017

Member

🎉

Member

skeggse commented Aug 9, 2017

🎉

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Aug 9, 2017

Member

Added a .npmignore in a8e7183 and published!

Member

LewisJEllis commented Aug 9, 2017

Added a .npmignore in a8e7183 and published!

@skeggse

This comment has been minimized.

Show comment
Hide comment
@skeggse

skeggse Aug 9, 2017

Member

The other option there, @LewisJEllis, is to specify a whitelist via the files array in the package.json.

Member

skeggse commented Aug 9, 2017

The other option there, @LewisJEllis, is to specify a whitelist via the files array in the package.json.

@LewisJEllis

This comment has been minimized.

Show comment
Hide comment
@LewisJEllis

LewisJEllis Aug 9, 2017

Member

Oh, nice, TIL. That seems way easier given that I started with a whitelist in the comments of the .npmignore.

Member

LewisJEllis commented Aug 9, 2017

Oh, nice, TIL. That seems way easier given that I started with a whitelist in the comments of the .npmignore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment