Can't read from multiple databases

[paco-cabrera]

I have an issue accessing multiple databases allocated in the same host one after another.

The thing is that I want to retrieve two different types of documents from two different databases with different authentication settings.

I have set up this example code.

from docs.doc1 import Doc1
from docs.doc2 import Doc2
from mongoengine import connect

db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='',
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)

Doc1._meta['db_alias'] = 'db1'
Doc2._meta['db_alias'] = 'db2'

connect(**db1_connection_dict)
connect(**db2_connection_dict)

read1_db1 = Doc1.objects().first()  # This works
read1_db2 = Doc2.objects().first()  # This works

read2_db1 = Doc1.objects().first()  # This crashes

I get an error as if the authorization wasn’t right.

I have inquired into this matter and the problem is that when it tries to query db1 for the second time, it is trying to access it with the db2 authentication settings.
I have realized that by setting up a breakpoit before the second query to the first database and running connectionStatus command in the db1 database:

import mongoengine
mongoengine.connections._dbs[‘db1’].command(‘connectionStatus’)

And I get
{'authInfo': {'authenticatedUsers': [{'user': 'db2', 'db': 'admin'}], 'authenticatedUserRoles': [{'role': 'read', 'db': 'db2'}]}, 'ok': 1.0}
Which are the db2 authentication settings.

If anyone could shed some light over this I would really appreciate it.

Thanks!

[bagerard]

Hi @paco-cabrera,
Initially I thought it could be related to mongodb version but I just tried with 3.4 and 3.6 and I get the expected results by executing a code similar to yours.

By first loading 1 empty document in db1 with the following:

import pymongo
uri = "mongodb://db1:db1@127.0.0.1/?authSource=admin"
client = pymongo.MongoClient(uri)
client.db1.doc1.drop()
client.db1.doc1.insert({})

uri = "mongodb://db2:db2@127.0.0.1/?authSource=admin"
client = pymongo.MongoClient(uri)
client.db2.doc2.drop()

If I then run the following:

import mongoengine
db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='',
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)

class Doc1(mongoengine.Document):
    meta = {'db_alias': 'db1'}

class Doc2(mongoengine.Document):
    meta = {'db_alias': 'db2'}

mongoengine.connect(**db1_connection_dict)
mongoengine.connect(**db2_connection_dict)

assert isinstance(Doc1.objects().first(), Doc1)
print(mongoengine.connection._dbs['db1'].command('connectionStatus'))    # {u'authInfo': {u'authenticatedUsers': [{u'db': u'admin', u'user': u'db1'}], u'authenticatedUserRoles': [{u'db': u'admin', u'role': u'readWriteAnyDatabase'}, {u'db': u'admin', u'role': u'dbAdminAnyDatabase'}, {u'db': u'admin', u'role': u'userAdminAnyDatabase'}]}, u'ok': 1.0}
assert Doc2.objects().first() is None
print(mongoengine.connection._dbs['db1'].command('connectionStatus'))    # {u'authInfo': {u'authenticatedUsers': [{u'db': u'admin', u'user': u'db2'}], u'authenticatedUserRoles': [{u'db': u'admin', u'role': u'readWriteAnyDatabase'}, {u'db': u'admin', u'role': u'dbAdminAnyDatabase'}, {u'db': u'admin', u'role': u'userAdminAnyDatabase'}]}, u'ok': 1.0}

assert isinstance(Doc1.objects().first(), Doc1)
assert Doc2.objects().first() is None

The assertions don't go in error (thus mongoengine reads at the right place) but its true that the output of connectionStatus is strange. What happens behind the scene in mongoengine is that it uses the same MongoClient (as both dbs are in the same host) and then authenticates in the 2 different databases through this same client. This is allowed by pymongo. It looks like the output of connectionStatus gives you the latest authinfo but I don't think there is anything wrong there

[paco-cabrera]

Hi @bagerard,
First of all thank you for the reply and the time you spent trying to solve the issue.

But I have run the same exact code you posted in your reply and I'm still getting the same authentication error when trying to do the second query to db1:
OperationFailure: not authorized on db1 to execute command { find: "doc1", filter: {}, limit: 1, singleBatch: true, $db: "db1", $readPreference: { mode: "primaryPreferred" } }

There's a thing I shall comment, mongod security is enabled in my localhost, so you have to be correctly authorized before accessing any database.
By default local host mongo daemons have the security disabled, so maybe yours is deactivated too. I'm saying this because a colleague of mine said it worked, but when he turned on the authorization it crashed.
If want to give it a try you have to open the config file of the mongo daemon (if you have ubuntu by default it is in /etc/mongod.config) and look for this lines, that most likely will be commented:

security:
  authorization: "enabled"

So if you want to replicate the error, you can follow this steps:
Create 'db1' and 'db2' users in admin db with 'readWrite' roles in the respective databases, edit the mongod.config uncomment the security lines (enable authorization), stop the mongo daemon and restart it.

Most surely the code you wrote above will crash, when trying to query 'db1' for the second time.

If you try this, and have any thoughts on the matter I'd love to hear about them.

Thank you.

[bagerard]

Hi, I had security turned on during my test but both my users had readWriteAnyDatabase role so that's why it was working :).

If I set up the users' roles properly, I now see the exact same error as you.

The error relates to this, which is +- what's happening behind mongoengine:

import pymongo
client = pymongo.MongoClient()

client.db1.authenticate('db1', 'db1', source='admin')
client.db1.doc1.find_one()
client.db2.authenticate('db2', 'db2', source='admin')   # overwrite auth
client.db1.doc1.find_one()

When the host is the same, it uses the same MongoClient and then uses the .authenticate method of pymongo.Database to authenticate but it looks like the last .authenticate call overwrite the previous one.

pymongo.Database.authenticate method actually got deprecated in pymongo and the doc mentions that

.... Instead of using these methods to change credentials, pass credentials for one user to the MongoClient at construction time, and either grant access to several databases to one user account, or use a distinct client object for each user.

To fix this we would need to either:

• use a different client for every alias
• make it smart enough to understand better if it should create a new client or if it could reuse the existing one

WORKAROUND
mongoengine checks both host key of your connection settings to identify if it need to create an additional client. So if both your database are running locally, changing your settings to the following tricks mongoengine:

db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='127.0.0.1',    # changed
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)

[paco-cabrera]

Hello @bagerard!
I'm kind of glad that you found the error as I was beginning to think I might be going crazy or something :).

Thanks for the explanation as I understand now where does the error come from.

I came up with the error when connecting to remote mongolatlas dbs hosts, but I have been able to implement your workaround by reordering the list of hosts in one connection; so thanks for that too.

Anyway, is this going to be fixed in the next update of the code?
If so, let me know if I can contribute in some way, because using the workaround is fine for now, but I think would be nicer if it worked in a proper way.

Again, thank you for everything!

[bagerard]

You are welcome. I'm glad we figured it out. I started working on this, it will be fixed in the next version