Tdl 23543 fix uuid transformation #100

RushiT0122 · 2023-07-18T14:15:55Z

Description of change

Fix the uuid transformation issue because UUIDLegacy support was removed by PyMongo v4.3+ and reads UUID fields as bson.binary.Binary object causes the transform issue.
Fix multiple compatibility issues in integration tests.

QA steps

automated tests passing

On local env, tested compatibility with,

MongoDB v4.4
MongoDB v.5.0
MongoDB v6.0

Risks

Rollback steps

revert this branch

- sort the dictionary before assertion

RushiT0122 · 2023-07-18T17:41:44Z

Since Pymongo v4.4.0 is incompatible with python version 3.5.6 currently CCi is failing. We will be able to run it end-2-end once we merge Orchestrator PR#292.

…oddb v6.0

- improve integration testlogging

bhtowles · 2023-07-20T13:35:42Z

tests/test_mongodb_incremental_open_transactions.py

@@ -306,3 +330,8 @@ def test_run(self):

            # validate that the state value has not changed after deleting the bookmarked value in each collection
            self.assertEqual(state_2, state_3)
+
+            # Reset the transaction life time limit
+            LOGGER.info(f"Re-setting transaction life time limit to {original_life_time_limit} seconds")


I've been informed that using f strings in LOGGER will fail pylint so I've been using:
LOGGER.info("Text %s", variable) format now

I have not seen this issue. Also AFAIK pylint doesn't run on tests directory.

Correct, it only runs on ui-automation / webapp side. But I try to be consistent across both

Removed the f-string and replaced it with .format().

tests/test_mongodb_projection.py

luandy64 · 2023-07-25T18:02:03Z

Makefile

@@ -1,4 +1,4 @@
 .DEFAULT_GOAL := test

 test:
-	pylint tap_mongodb tap_mongodb/sync_strategies -d missing-docstring,fixme,duplicate-code,line-too-long,too-many-statements,too-many-locals,consider-using-f-string,consider-using-from-import
+	pylint tap_mongodb tap_mongodb/sync_strategies -d missing-docstring,fixme,duplicate-code,line-too-long,too-many-statements,too-many-locals,consider-using-f-string,consider-using-from-import,,broad-exception-raised,superfluous-parens,consider-using-generator


Looks like there is an extra comma here. I'm a little shocked pylint doesn't complain about this 😂

luandy64 · 2023-07-25T18:04:06Z

tests/mongodb_common.py

@@ -1,5 +1,6 @@
 import os
 import pymongo
+from tap_tester.logger import  LOGGER


Suggested change

from tap_tester.logger import LOGGER

from tap_tester.logger import LOGGER

luandy64 · 2023-07-25T18:14:38Z

tests/test_mongodb_incremental_open_transactions.py

+            # Reset the transaction life time limit
+            LOGGER.info("Re-setting transaction life time limit to {} seconds".format(original_life_time_limit))
+            success, _ = self.set_transaction_life_time_limit_seconds(client, original_life_time_limit)
+            self.assertTrue(success, msg="Failed to reset transaction life time limit")


Does it make sense to fail the test if all test assertions pass, but the clean up fails?

We are making changes to the database-level configuration. If we don't reset it then the remaining integration tests will run with the updated configuration, potentially causing us to miss critical issues.

luandy64

Approving assuming the tests pass

1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values.

- sort the dictionary before assertion

…oddb v6.0

- improve integration testlogging

…-io/tap-mongodb into TDL-23543-fix-uuid-transformation

* Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

…sformation

* projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com>

* Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Tdl 23543 fix uuid transformation (#100) * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * [Tdl 23471] Fix circle ci config (#101) * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> * Tdl 23609 fix empty projection (#102) * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> * renamed build_mongo_4_4 to build in config.yml * Renamed build_mongo_4_4 to build in config.yml --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

* Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Major version bump 3.0.0 * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Tdl 23543 fix uuid transformation (#100) * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * [Tdl 23471] Fix circle ci config (#101) * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> * Tdl 23609 fix empty projection (#102) * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> * Removed CCi changes from CHANGELOG.md * renamed build_mongo_4_4 to build in config.yml * Renamed build_mongo_4_4 to build in config.yml * Major version bump 3.0.0 * Removed CCi changes from CHANGELOG.md --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

RushiT0122 added 5 commits July 17, 2023 12:59

- set maxDiff to None to reset dict comparison limit

8def8a6

- sort the dictionary before assertion

fix uuid transformation

1354227

upgrade pymongo version in cci configuration

05672e9

fix cci configuration

1b53d6f

upgrade pip and setuptools

22ac45c

RushiT0122 force-pushed the TDL-23543-fix-uuid-transformation branch from 3d904b8 to 22ac45c Compare July 18, 2023 14:38

RushiT0122 changed the base branch from master to TDL-22154-upgrade-pymongo July 18, 2023 16:00

RushiT0122 changed the base branch from TDL-22154-upgrade-pymongo to master July 18, 2023 17:16

RushiT0122 changed the title ~~Tdl 23543 fix UUID transformation~~ Tdl 23543 fix uuid transformation Jul 18, 2023

RushiT0122 changed the base branch from master to TDL-22154-upgrade-pymongo July 18, 2023 18:30

RushiT0122 added 7 commits July 18, 2023 18:51

update cci config to run on latest python version

2f8ec23

fix linting issues

e58034d

fix uuid bookmarking

da4d6fa

fix integration tests

6fb9087

remove debugger import

345aeaf

fix integration test compatibility issues with pymongo v4.3+ and mong…

0e8b389

…oddb v6.0

- fix transaction life time limit

9ec305e

- improve integration testlogging

RushiT0122 requested review from bhtowles and luandy64 July 20, 2023 10:45

bhtowles reviewed Jul 20, 2023

View reviewed changes

tests/test_mongodb_projection.py Outdated Show resolved Hide resolved

RushiT0122 added 2 commits July 20, 2023 18:00

fix typo

2ce7b7f

remove f-string from logger statements

cc1b3dc

luandy64 reviewed Jul 25, 2023

View reviewed changes

minor code cleanup

f9acfc1

luandy64 approved these changes Jul 25, 2023

View reviewed changes

shantanu73 and others added 3 commits July 26, 2023 11:47

Changes:

11b9c31

1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values.

- set maxDiff to None to reset dict comparison limit

70fcdab

- sort the dictionary before assertion

fix uuid transformation

965a07f

RushiT0122 and others added 17 commits July 26, 2023 11:47

upgrade pymongo version in cci configuration

a8bf4c6

fix cci configuration

1312980

upgrade pip and setuptools

d48f80b

update cci config to run on latest python version

ae64af7

fix linting issues

fbfff82

fix uuid bookmarking

91334b2

fix integration tests

a127740

remove debugger import

897061c

fix integration test compatibility issues with pymongo v4.3+ and mong…

1558dc8

…oddb v6.0

- fix transaction life time limit

26cb6ec

- improve integration testlogging

fix typo

2447691

remove f-string from logger statements

fb5d295

minor code cleanup

03de6dd

Merge branch 'TDL-23543-fix-uuid-transformation' of github.com:singer…

50389dc

…-io/tap-mongodb into TDL-23543-fix-uuid-transformation

Merge branch 'TDL-22154-upgrade-pymongo' into TDL-23543-fix-uuid-tran…

ff69d9f

…sformation

shantanu73 merged commit c900ebb into TDL-22154-upgrade-pymongo Jul 27, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tdl 23543 fix uuid transformation #100

Tdl 23543 fix uuid transformation #100

RushiT0122 commented Jul 18, 2023 •

edited

Loading

RushiT0122 commented Jul 18, 2023

bhtowles Jul 20, 2023 •

edited

Loading

RushiT0122 Jul 20, 2023 •

edited

Loading

bhtowles Jul 20, 2023

RushiT0122 Jul 21, 2023

luandy64 Jul 25, 2023

luandy64 Jul 25, 2023

luandy64 Jul 25, 2023

RushiT0122 Jul 25, 2023

luandy64 left a comment

	from tap_tester.logger import LOGGER
	from tap_tester.logger import LOGGER

Tdl 23543 fix uuid transformation #100

Tdl 23543 fix uuid transformation #100

Conversation

RushiT0122 commented Jul 18, 2023 • edited Loading

Description of change

QA steps

Risks

Rollback steps

RushiT0122 commented Jul 18, 2023

bhtowles Jul 20, 2023 • edited Loading

Choose a reason for hiding this comment

RushiT0122 Jul 20, 2023 • edited Loading

Choose a reason for hiding this comment

bhtowles Jul 20, 2023

Choose a reason for hiding this comment

RushiT0122 Jul 21, 2023

Choose a reason for hiding this comment

luandy64 Jul 25, 2023

Choose a reason for hiding this comment

luandy64 Jul 25, 2023

Choose a reason for hiding this comment

luandy64 Jul 25, 2023

Choose a reason for hiding this comment

RushiT0122 Jul 25, 2023

Choose a reason for hiding this comment

luandy64 left a comment

Choose a reason for hiding this comment

RushiT0122 commented Jul 18, 2023 •

edited

Loading

bhtowles Jul 20, 2023 •

edited

Loading

RushiT0122 Jul 20, 2023 •

edited

Loading