Cluster support for mongodb

[ghost]

Need a way to connect to a mongo cluster. The cluster has one writer and (in our case) 2 readers, so I can get seeding working by adjusting the IP of writer each time I load fixtures, but this is far from idea.

The url I'm using is of the form
mongodb://ip1:port1,ip2:port2,ip3:port3/db_name

The line in question seems to be:
https://github.com/mgan59/mongoose-fixture/blob/master/lib/MongooseFixtureCommand.js#L272

[mgan59]

Seems like an easy enough adjustment. Think the most sensible implementation would be to have MongooseFixtureCommand check if the mongoSettings from line #271 is an array instead of an object literal, and if so to deserialize the array of connection-object settings. That way the api supports both a single connection versus multiples, maintains a backward compatibility for those that use the object-literal notation. Make sense? Does that work for you?

[ghost]

absolutely - I do this for the current system:

var uri_from_arrays = function(hosts, ports, dbname) {
var url = '';
for(var i = 0; i < hosts.length; i++) {
if(url.length > 0) url += ',';
url += hosts[i] + ':' + ports[i];
}
return 'mongodb://' + url + '/' + dbname;
};

It supports single hosts and clusters.

[mgan59]

@roger-kaybus I began working on this fix. Let me know if the following mongoHostSetting config variable works for you?

// Create our fixture config with defined
// mongo-connection and file paths
var fixtureConfig = FixtureConfig({
    mongoConnection:{
        'servers':[
            {
                'host':'localhost',
                'port':'27010',
            },
            {
                'host':'localhost',
                'port':'27011',
            }
            // ... put additional hosts/posts
        ],
        'dbname':'mongoose-fixture-test'
    },
    paths:{
        schemaPath:__dirname+'/schemas/',
        dataFixturePath:__dirname+'/fixtures/'
    }
});

[mgan59]

Making a note here for replicast connections with mongoose

[http://mongoosejs.com/docs/connections.html#replicaset_connections](mongoosejs multi-conn replicaSet)

mongoose.connect('mongodb://username:password@host:port/database,mongodb://username:password@host:port,mongodb://username:password@host:port?options...' [, options]);

[ghost]

awesome. looks great.

[mgan59]

@roger-kaybus spent a little more time working on this and it turns out to be slightly more involved than I had originally thought. Will be a few more days before I have it completed.

[mgan59]

@roger-kaybus could I get a little more info from you. Are you trying to connect to a cluster meaning a replica-set? Or are you trying to connect to multiple mongo-instances with different dbs? Or are you trying to connect to a set of sharded mongo instances?

[ghost]

We're trying to connect to a 3 server replica set. There are two readers and one writer in the cluster, so only one instance can take writes.

Let me know if you need more info.

[mgan59]

@roger-kaybus I configured a 3 server replica set as part of my test suite and it appears that writing to the primary node using mongoose-fixture does in fact cascade the writes to the read machines fairly quickly. Are you configuring the replica set correctly? This was a great resource https://www.coffeepowered.net/2010/08/06/setting-up-replica-sets-with-mongodb-1-6/

[ghost]

the issue was the primary changed from day to day without a failure. one day primary would be server1, next its server2 without any event on the cluster.

thats how we initially worked - just change the host to point to the primary every time it changed. but it changed very often. leave your cluster for a day or two any try again. the primary will change.

[mgan59]

mgoose-replica:PRIMARY> rs.status().members[0].uptime / (60 * 60)
15.21111111111111

K I'm at 15hours will keep checking :)

Did you consider adjusting the priority levels? Maybe there is a brief interruption of service triggers an election that puts the secondary as primary and with no priority levels when primary does come backup an election triggers but there is no weight applied to force the election in favor of the originally primary node.

Now that I understand your problem more fully the approach needs to change. I'll see what I can do later this evening.

[mgan59]

@roger-kaybus good news I have a working version on 10-cluster-support. I essentially refactored the db connection system, before I was using the mongoose singleton as my proxy to the db connection. Now I use the mongoose.createConnection interface which returns a mongodb connection object which is then passed into the FixtureLoader. This will create an api breakage for all users as the data-fixture signature now requires an additional parameter.

As a result this change is going into a 0.3.0 release and the 0.2.x line will be closing.

I have the tests written and have kicked the tires, but there remains some more administrative tasks before the release can occur. I'll let you know when the new version is on npm most likely this weekend. If you are able to pull my branch and give it a test before hand that would be great.

[ghost]

i'll pull the branch and try it. thanks

[mgan59]

completed work on 0.3.0 and merged the branch with this changeset

let me know how things go :)

[mgan59]

oh and the new version is on npm

[ghost]

cool. i'll check out the npm today.

[ghost]

fyi - did an npm install --save mongoose-fixture and package.json says "mongoose-fixture": "~0.2.3". looks like the npm repo is not updated.

[ghost]

but https://npmjs.org/package/mongoose-fixture says 0.3.0. Let me investigate.

[ghost]

npm install -save mongoose-fixture@0.3.0 worked.

[ghost]

working good now with single and cluster env with one issue. I'll open a separate ticket.