Async mongodb driver by allanbank · Pull Request #180 · brianfrankcooper/YCSB

allanbank · 2014-06-20T03:11:16Z

This pull request contains a new driver for MongoDB based on the asynchronous driver. The bulk of the changes are adding the required logic to the startup scripts to support the second driver. I have used this branch to run a series of benchmarks.

There are also a number of community contributed updates to the old MongoDB Driver. These include:

Updated README with specific instructions for running a MongoDB suite.
Updates to the configuration to now be strictly based on a MongoDB Connection String URI which is standardized across drivers.
Other small fixes and performance enhancements.

This pull request will supersede the following pull requests:

PR-112 - Integrates the MongoDB Connection String URI for both drivers.
PR-115 - The writeConcern and readPreference are expressed via the standardized MongoDB Connection String URI.
PR-131 - Read preferences are passed via the MongoDB Connection String and replica sets are automatically detected for the asynchronous driver and will be enabled with the MongoDB Inc. driver by passing multiple servers in the URI.

I know this is a large volume of changes but they are all focused on the MongoDB driver and thought it would be easier to provide them in one batch than a bunch of pull requests. If you would prefer a bunch of smaller changes I will need guidance on the desired granularity.

allanbank · 2014-06-21T03:43:05Z

Note says that the Travis Build Failed but it appears to be an issue with Travis not being able to download artifacts from Maven. Can someone trigger a manual rebuild?

westonplatter · 2014-06-21T03:45:43Z

@allanbank I bumped if for a rebuild (even though the Github status does not show the reubuild).

busbey · 2015-05-20T13:48:19Z

I restarted the build now that the mapkeeper stuff is out of the way. I'd like to get this set of changes merged in before cutting the next release.

If someone from the mongo community can work with me on needed updates, I'll proactively go point the related open issues here and close them.

busbey · 2015-05-20T13:50:40Z

the java home stuff is good, but seems unrelated to the rest of the mongo improvements. If I file an issue for "allow the use of java home" could you edit the commit message to call out closing it?

I wrote an issue: #252.

great. presuming you go forward with the rebase, please edit the commit message for this part of the changes to note "fixes #252"

allanbank · 2015-05-20T15:14:33Z

@busbey - Give me a day or two to make the changes you suggested. Is it OK if I rebase the pull on the latest master making the changes as I go?

busbey · 2015-05-21T22:02:19Z

please feel free to rebase. I prefer rebases to lots of merges, personally. :)

Allow setting the java version to use via the "JAVA_HOME" environment variable. Fixes brianfrankcooper#252.

…level will throw exceptions and be caught in catch block

example

Changed reference to original forked repo which is not maintained to this repo to avoid conflicting instructions. Might want to do that for top level README as well.

When running workloads that use the "insert" operation, we get a lot of duplicate key exceptions from the server. Changing to upsert to avoid these exceptions.

URI to configure the client. Old properties are deprecated and but will be added to the URI if present and the URI does not already contain a value for the setting.

to the 3.0 driver from MongoDB Inc. and updated to the new Document and fluent APIs.

allanbank · 2015-05-22T04:26:14Z

@busbey -

I think that latest push covers all of the issues you raised. Let me know if I missed something.

Rob.

busbey · 2015-05-23T16:22:06Z

The update looks great.

It looks like with the current update we'll be using the Mongo 3 client, right? Will that prevent us from talking to Mongo 2 servers? If so, how comon are Mongo 2 installations? Is it worth maintaining both, ala our HBase binding?

allanbank · 2015-05-23T21:16:39Z

The driver can talk to version 2 and 3 clusters.

Rob

On May 23, 2015 12:22:12 PM EDT, Sean Busbey notifications@github.com wrote:

The update looks great.

It looks like with the current update we'll be using the Mongo 3
client, right? Will that prevent us from talking to Mongo 2 servers? If
so, how comon are Mongo 2 installations? Is it worth maintaining both,
ala our HBase binding?

Reply to this email directly or view it on GitHub:
#180 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Mongodb] merge adds Async mongodb driver, update normal driver to talk to mongo 2 and 3 clusters.

busbey · 2015-05-24T05:23:04Z

Thanks for sticking with this contribution! I hope to see more great stuff from the mongo community in the future.

allanbank · 2015-05-27T01:06:47Z

@busbey - No problem. Thanks for doing the merge.

I mentioned it in the initial pull request but I am pretty certain these pull requests can be closed now.

PR-112 - Integrates the MongoDB Connection String URI for both drivers.
PR-115 - The writeConcern and readPreference are expressed via the standardized MongoDB Connection String URI.
PR-131 - Read preferences are passed via the MongoDB Connection String and replica sets are automatically detected for the asynchronous driver and will be enabled with the MongoDB Inc. driver by passing multiple servers in the URI.

If you would like me to add an explicit comment to each of those explaining why it can be closed I'll be happy to do that.

Rob.

busbey · 2015-05-27T06:32:12Z

Thanks for the reminder, I'll go close them out.

…driver [Mongodb] merge adds Async mongodb driver, update normal driver to talk to mongo 2 and 3 clusters.

busbey reviewed May 20, 2015
View reviewed changes

allanbank and others added 18 commits May 22, 2015 00:07

Add an adapter for MongoDB using the asynchronous driver.

1db6c0c

Add the async client to the startup script.

7e0f632

Allow setting the java version to use via the "JAVA_HOME" environment variable. Fixes brianfrankcooper#252.

Optimizations.

8e00106

Add option: threadsAllowedToBlockForConnectionMultiplier

878f9f9

Update MongoDB Async Driver version.

a3bbb25

Cleanup

218e258

Don't use finalizers to close cursors

91f0791

Updated to current standard write concerns

7b9cf5f

No need to call getError on writeResult. any errors for writeConcern …

cd1fb35

…level will throw exceptions and be caught in catch block

Set default thread count to 100 when thread count is not specified

aa1e56a

Added 'Install Maven and Java' instructions Ycsb config parameters

488ef7f

example

Add maven bash env vars

637531b

add instructions to install java via yum/apt-get

c960961

added support to specify readPreference

0b68bd4

Update mongodb/README.md with more specific info

842560b

Changed reference to original forked repo which is not maintained to this repo to avoid conflicting instructions. Might want to do that for top level README as well.

Revert to the central YCSB repository.

7762b5e

Change inserts to upserts in mongoclient

40bdb50

When running workloads that use the "insert" operation, we get a lot of duplicate key exceptions from the server. Changing to upsert to avoid these exceptions.

README Tweaks.

cdecd99

allanbank added 10 commits May 22, 2015 00:08

Update the README with how to use the asynchronous driver.

03b2530

Update the MongoDB drivers to just use a connection string or MongoDB

3b9e207

URI to configure the client. Old properties are deprecated and but will be added to the URI if present and the URI does not already contain a value for the setting.

Add the avoidance of finalizers for cursors back in.

6ea36c1

Remove eclipse-ism.

4bcbb59

Add logback logger.

7c24d19

Explicit License.

61b9694

Explicit License.

12ddd5d

Bump the MongoDB versions.

d562611

Tests and updates for OptionsSupport.

eaebc71

Add majority write concern and deprecated most of the options. Migrated

6970ee1

to the 3.0 driver from MongoDB Inc. and updated to the new Document and fluent APIs.

allanbank force-pushed the async-mongodb-driver branch from a243b47 to 6970ee1 Compare May 22, 2015 04:16

busbey added a commit that referenced this pull request May 24, 2015

Merge pull request #180 from allanbank/async-mongodb-driver

0ba7e79

[Mongodb] merge adds Async mongodb driver, update normal driver to talk to mongo 2 and 3 clusters.

busbey merged commit 0ba7e79 into brianfrankcooper:master May 24, 2015

This was referenced May 27, 2015

Use the new MongoClient constructor #112

Closed

MongoDB readPreference and writeConcern improvements #115

Closed

Added support of testing MongoDB replica set with automatic failover. #131

Closed

allanbank deleted the async-mongodb-driver branch May 28, 2015 01:49

jaricftw pushed a commit to jaricftw/YCSB that referenced this pull request Jul 19, 2016

Merge pull request brianfrankcooper#180 from allanbank/async-mongodb-…

627d3f7

…driver [Mongodb] merge adds Async mongodb driver, update normal driver to talk to mongo 2 and 3 clusters.

jaricftw pushed a commit to jaricftw/YCSB that referenced this pull request Jul 19, 2016

Merge pull request brianfrankcooper#180 from allanbank/async-mongodb-…

f175e9e

…driver [Mongodb] merge adds Async mongodb driver, update normal driver to talk to mongo 2 and 3 clusters.

Conversation

allanbank commented Jun 20, 2014

Uh oh!

allanbank commented Jun 21, 2014

Uh oh!

westonplatter commented Jun 21, 2014

Uh oh!

busbey commented May 20, 2015

Uh oh!

busbey May 20, 2015

Choose a reason for hiding this comment

Uh oh!

allanbank May 20, 2015

Choose a reason for hiding this comment

Uh oh!

busbey May 21, 2015

Choose a reason for hiding this comment

Uh oh!

allanbank commented May 20, 2015

Uh oh!

busbey commented May 21, 2015

Uh oh!

allanbank commented May 22, 2015

Uh oh!

busbey commented May 23, 2015

Uh oh!

allanbank commented May 23, 2015

Uh oh!

busbey commented May 24, 2015

Uh oh!

allanbank commented May 27, 2015

Uh oh!

busbey commented May 27, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants