Querys and ReferenceField errors with legacy MongoDB dataset and custom primary_key fields

[sramko]

Hi,

I'm a first time mongoengine user and not sure, if this is a bug or I'm not using the system correctly.

I have a existing and filled mongo-database, which is currently holding information from a former relational database. We are currently using mongoDB to transform and clean this data, prior to migrating it to a new system.
I'm not trying to build a MongoEngine Object mapping on this existing data. But it fails to handle the custom 'primary_key' setting together with ReferenceField-field-types.
The former primary key from the RDB are all in the MongoDB, too.

I've tried using 'Document' and 'DynamicDocument' with the same results.
I've tried as well using "primary_key=True, DBRef=False", but the "DBRef" parameter has no influence on the results.

When not defining the Int32 fields as primary key, thank I can query for 'Country.Code' without error

MongoEngine: 0.23.0
PyMongo: 3.11.3
MongoDB: 4.4.4. CE
Python: 3.9.2

from mongoengine import *

# This is for an already existing and entry-filled collections,
# 'Code' is filled with unique int32 values and was a PK in a previous relational DB
class Country(Document):
   meta = {'db_alias': 'alias',\
       'collection': 'some_cryptic_country_collection_name'}
   Code = IntField(primary_key=True)
   Description = StringField()

# This is for an already existing and entry-filled collections,
# 'EntryCode' is filled with unique int32 values and was a PK in a previous relational DB
# 'EntryCode' is filled with int32 values and was a foreign key to Country-entries in a previous relational DB
class Entry(Document):
   meta = {'db_alias': 'alias',\
       'collection': 'some_cryptic_entry_collection_name'}
   EntryCode = IntField(primary_key=True)
   CountryCode = ReferenceField('Country')
   EntryName = StringField()

connect( \
 host=getConfig().getMongoDB_MongoHost(), \
 port=getConfig().getMongoDB_MongoPort(), \
 username=getConfig().getMongoDB_MongoUser(), \
 password=getConfig().getMongoDB_MongoPasswod(), \
 db = 'database', \
 alias='alias', \
 authentication_source='admin')

def _testMongoEngineDBConnection():
   print("### TEST MongoEngineDBConnection:")
   country = Country.objects[0]
   print("country: " + country.Description + "; Code: " + str(country.Code)) # works, provides the correct output
   # works fine as well. It finds the entry with the Code/PK 2349 and prints the information
   for country in Country.objects:
       if country.Code == 2349:
           print("country 2349: " + country.Description + "; Code: " + str(country.Code)+ "; pk: " + str(country.pk) ) 

   # 1st Error: No object with ID=2349 found
   if Country.objects(id=2349).first() == None:
       print("ERROR: no country objects with id 2349  found")
   else:
       country = Country.objects(id=2349)
       print("country id 2349: " + country.Description)

   # 2nd Error: No object with PK=2349 found
   if Country.objects(pk=2349).first() == None:
       print("ERROR: no country objects with pk 2349  found")
   else:
       country = Country.objects(pk=2349)
       print("country pk 2349: " + country.Description)
 
   # 3rd Error: No object with Code=2349 found
   # I've tried as well 'Code', and 'Code__' and Code__
   if Country.objects(Code=2349).first() == None:
       print("ERROR: no country objects with code 2349  found")
   else:
       country = Country.objects(pk=2349)
       print("country pk 2349: " + country.Description)

   # Searching for Description works just fine (2349 is 'Japan' in the dataset)
   if Country.objects(Description__='Japan').first() == None:
       print("ERROR: no country objects with Description='Japan' found")
   else:
       country = Country.objects(Description__='Japan').first()
       print("country Description='Japan': " + country.Description + "; Code: " + str(country.Code))

   # Usind a raw-search works as well fine
   if Country.objects(__raw__={'Code': 2349}).first() == None:
       print("ERROR: no country objects with raw code 2349  found")
   else:
       country = Country.objects(__raw__={'Code': 2349}).first()
       print("country raw code 2349: " + country.Description)

   # 4th Error: loading a random 'Entry' object and loading the referenced 'Country' object results in a error
   # The Entry-object in this case is referencing to the Country-ID 2349
   if Entry.objects.first() == None:
       print("ERROR: no Entry objects found")
   else:
       entry = Entry.objects[0]
       print("entry : " + company.EntryName)
       #... and the next line results in the Error
       print("entry's country.description: " + entry.CountryCode.Description)

_testMongoEngineDBConnection()

country: SomeCountry; Code: 123
country 2349: Japan; Code: 2349; pk: 2349
ERROR: no country objects with id 2349  found
ERROR: no country objects with pk 2349  found
ERROR: no country objects with code 2349  found
country Description='Japan': Japan; Code: 2349
country raw code 2349: Japan
entry : SomeEntryName

ERROR  [...Trace...]
  File "C:\Users\xyz\AppData\Local\Programs\Python\Python39\lib\site-packages\mongoengine\fields.py", line 1215, in __get__
    raise DoesNotExist("Trying to dereference unknown document %s" % value)
mongoengine.errors.DoesNotExist: Trying to dereference unknown document DBRef('some_cryptic_country_collection_name', 2349)

[bagerard]

Closing this, issue is unclear, please probide a reproducible snippet if you want to have a chance to get more assistance