MongoDB (v2 - Java) connector is not extracting the raw data from the database

[prconstruct]

Enviroment

• Airbyte version: 0.29.21 alpha
• OS Version / Instance: Ubuntu 20.04 lts
• Deployment: Docker (in local)
• Source Connector and version: airbyte/source-mongodb-v2:0.1.0
• Destination Connector and version: airbyte/destination-s3:0.1.12 (JSON lines)
• Severity: Critical
• Step where error happened: Sync job

Current Behavior

I'm currently facing issues with Source MongoDB (v2 - java version) connector as the data is not exported as raw/same as in DB. The connector converts nested object columns to string and an array of objects are converted as Document{}. Since this is a critical source for us, we want this to rectify asap. Also in the meantime, I have planned to develop our own custom connector in python since I'm not proficient in Java.

MongoDB:
{"_id": "61431b7c9f28c41b367d5205", "index": 8, "guid": "812e3e02-b468-4842-b20e-d05c89166302", "isActive": true, "balance": "$1,207.39", "picture": "http://placehold.it/32x32", "age": 26, "array_col": ["adipisicing", "amet", "dolor", "consequat", "esse", "laboris", "dolor"], "object_array_col": [{"id": 0, "name": "Hunter Curry"}], "object_col": {"id": 8, "wording": "nulla"}, "greeting": "Hello, undefined! You have 10 unread messages.", "favoriteFruit": "banana", "updatedAt": "2021-09-08T06:45:37.431000Z"}

Airbyte:
{"_id":"61431b7c9f28c41b367d5205","index":8,"guid":"812e3e02-b468-4842-b20e-d05c89166302","isActive":true,"balance":"$1,207.39","picture":"http://placehold.it/32x32","age":26,"array_col":"[adipisicing, amet, dolor, consequat, esse, laboris, dolor]","object_array_col":"[Document{{id=0, name=Hunter Curry}}]","object_col":"{\"id\": 8, \"wording\": \"nulla\"}","greeting":"Hello, undefined! You have 10 unread messages.","favoriteFruit":"banana","updatedAt":"2021-09-08T12:15:37Z"}

Expected Behavior

The expected result is to export the actual data from MongoDB without doing any datatype conversion.
Expected Airbyte record:
{"_id": "61431b7c9f28c41b367d5205", "index": 8, "guid": "812e3e02-b468-4842-b20e-d05c89166302", "isActive": true, "balance": "$1,207.39", "picture": "http://placehold.it/32x32", "age": 26, "array_col": ["adipisicing", "amet", "dolor", "consequat", "esse", "laboris", "dolor"], "object_array_col": [{"id": 0, "name": "Hunter Curry"}], "object_col": {"id": 8, "wording": "nulla"}, "greeting": "Hello, undefined! You have 10 unread messages.", "favoriteFruit": "banana", "updatedAt": "2021-09-08T06:45:37.431000Z"}

Steps to Reproduce

1. Create a collection and Insert the provided source record in MongoDB
2. Configure source, destination, and connector
3. Run the connector or run docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/{workspace} --network host --log-driver none airbyte/source-mongodb-v2:0.1.0 read --config source_config.json --catalog source_catalog.json

Are you willing to submit a PR? Since I'm not proficient in Java, I'm unable to figure the issue in code.

[sherifnada]

@prconstruct thanks for creating the issue report - we'll address this in the second half of October. The MongoDB connector is currently made to be very "simple" just to shuttle data.

For the mean time is it an option to transform the data in the destination using JSON parsing/string manipulation functions? It may be a lot easier than working through creating a new connector

[prconstruct]

@sherifnada : Thanks. Yes, I can convert the string to JSON. But handling the "object_array_col" would be challenging since it was enclosed with Document{{ }}. I'm not able to find the appropriate type of this object in python.

[sherifnada]

@prconstruct could you upgrade to the latest version and see if the issue persists?

[prconstruct]

@sherifnada : Thanks a lot! It works fine as expected. Thanks a lot @irynakruk !

Result:
{"_airbyte_ab_id":"08e16042-32e3-42b0-a46b-9ce58203f58c","_airbyte_emitted_at":1635232150641,"_airbyte_data":{"_id":"61431b7c9f28c41b367d5205","index":8,"guid":"812e3e02-b468-4842-b20e-d05c89166302","isActive":true,"balance":"$1,207.39","picture":"http://placehold.it/32x32","age":26,"array_col":["adipisicing","amet","dolor","consequat","esse","laboris","dolor"],"object_array_col":[{"id":0,"name":"Hunter Curry"}],"object_col":{"id":8,"wording":"nulla"},"greeting":"Hello, undefined! You have 10 unread messages.","favoriteFruit":"banana","updatedAt":"2021-09-08T12:15:37Z"}}

[sherifnada]

amazing! glad to hear it