Source MongoDB: Support TLS

[sherifnada]

Tell us about the problem you're trying to solve

I would like to be able to connect to my Mongo DB cluster with TLS.

Describe the solution you’d like

I would like to have a checkmark/toggle in the connector setup page which controls whether or not I can connect via TLS.

This is doable as described in the Ruby Driver docs: https://docs.mongodb.com/ruby-driver/v2.2/tutorials/ruby-driver-create-client/

by setting the ssl option to true.

To implement this one needs to do the following:

(all files are in the airbyte-integrations/connectors/source-mongodb directory)

1. Add this as a boolean flag to spec.json
2. Edit lib/mongodb_source.rb to look at this boolean value and pass in the correct flag accordingly
3. Write a test case for this behavior

Describe the alternative you’ve considered or used

Not using Airbyte's connector

[nathan-protempo]

Mongodb Atlas requires TLS connections, so we can't use Airbyte with Mongodb Atlas until it supports TLS. https://docs.atlas.mongodb.com/reference/faq/security/

[sherifnada]

@nathan-protempo totally agreed that this is something we should have. Will get it done soon.

@arhip11 can we do the following to unblock this work:

1. create a Mongodb Atlas account
2. create an TLS-enabled Mongodb instance in cluster/replicaSet mode (see Source MongoDB: Support connecting to replicaSets #2421)
3. Share the credentials on lastpass

This will allow us to implement this issue in addition to #2421

To implement this issue, we'll need to:

1. populate the instance created above with some data. The DB doesn't need to have any significant amount of data in it. It only needs to have at least 1 collection (mongo's version of table) and some records in the collection.
2. add an integration test which connects to this DB using the TLS=true option and verifies that it can pull data

[arhip11]

@arhip11 can we do the following to unblock this work:

1. create a Mongodb Atlas account

Done

2. create an TLS-enabled Mongodb instance in cluster/replicaSet mode (see Source MongoDB: Support connecting to replicaSets #2421)

Cluster should be TLS-enabled by default. There is no explicit option to specify it.

3. Share the credentials on lastpass

Done.

To implement this issue, we'll need to:

1. populate the instance created above with some data. The DB doesn't need to have any significant amount of data in it. It only needs to have at least 1 collection (mongo's version of table) and some records in the collection.

Loaded sample dataset provided by vendor.

2. add an integration test which connects to this DB using the TLS=true option and verifies that it can pull data

@sherifnada - should we pass item 2 to the devs?

[yevhenii-ldv]

@sherifnada, I am having difficulty working on this task. I added ssl to the spec, and to test it, I connected to Ruby's environment and ran the check command. The check was successful pass, but the discover method constantly gives an error:

{"log":{"level":"INFO","message":"Discovering stream first_fake_collection"},"type":"LOG"}
/home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/result.rb:343:in `raise_operation_failure': CMD_NOT_ALLOWED: mapReduce (8000) (on cluster0-shard-00-05.iqgf7.mongodb.net:28017) (Mongo::Error::OperationFailure)
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/map_reduce/result.rb:115:in `validate!'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:29:in `block (3 levels) in validate_result'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:96:in `add_server_diagnostics'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:28:in `block (2 levels) in validate_result'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:43:in `add_error_labels'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:27:in `block in validate_result'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:82:in `unpin_maybe'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/response_handling.rb:26:in `validate_result'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/executable.rb:46:in `block in execute'
        from <internal:kernel>:90:in `tap'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/executable.rb:45:in `execute'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/op_msg_or_command.rb:28:in `block in execute'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/server/connection_pool.rb:590:in `with_connection'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/server.rb:429:in `with_connection'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/operation/shared/op_msg_or_command.rb:26:in `execute'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/collection/view/map_reduce.rb:245:in `send_initial_query'
        from /home/ykurochkin/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/mongo-2.14.0/lib/mongo/collection/view/map_reduce.rb:72:in `each'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_stream.rb:67:in `map'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_stream.rb:67:in `discover_properties'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_stream.rb:36:in `discover'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_source.rb:44:in `block in discover'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_source.rb:42:in `map'
        from /home/ykurochkin/PycharmProjects/airbyte/airbyte-integrations/connectors/source-mongodb/lib/mongodb_source.rb:42:in `discover'
        from source.rb:16:in `public_send'
        from source.rb:16:in `<main>'

I tried to test with all sample_ tables, and created my own with only 1 record, but I still got this error.
I found similar problems on the forums, and perhaps this error is related to the Free account on Mongodb Atlas:

1. [*] MongoDB Atlas - Does not work with Free plan (mapReduce() function not allowed) ForestAdmin/lumber#184
2. https://ask.csdn.net/questions/2691792

[sherifnada]

https://stackoverflow.com/a/45679466

Mapreduce is not allowed in free tier at this moment. https://docs.atlas.mongodb.com/unsupported-commands

@yevhenii-ldv please:

1. coordinate with @arhip11 to upgrade to the paid m10 cluster tier to be able to run mapreduce commands.
2. Update mongodb docs to note in bold that it requires the ability to perform a mapReduce operation and therefore Atlas' free tier will not work with our connector.
3. create an issue indicating that we should replace the mapReduce command in our mongodb connector with something else if possible.

[yevhenii-ldv]

@sherifnada Okay, I'll do this

[arhip11]

@yevhenii-ldv, @sherifnada - hey guys, m10 cluster is ready.