[Tdl 23471] Fix circle ci config #101

shantanu73 · 2023-07-20T13:01:32Z

Description of change

Fixed CircleCI configuration
Added configuration for MongoDB 6.0
Removed configuration for MongoDB 4.2 as it has reached End of Life

QA steps

automated tests passing
manual qa steps passing (list below)

Risks

Low risk

Rollback steps

revert this branch

luandy64 · 2023-07-25T17:50:36Z

.circleci/config.yml

@@ -155,7 +166,7 @@ jobs:
            export STITCH_CONFIG_DIR=/tmp/${CIRCLE_PROJECT_REPONAME}
            source /usr/local/share/virtualenvs/tap-tester/bin/activate
            pip install pymongo==4.4.0
-            selected_tests=$(circleci tests glob "tests/*{data,interrupt,open,restrict,view}*")
+            selected_tests=$(circleci tests glob "tests/*{test_mongodb}*")


According to the docs, it seems like this can be simplified to just *test_mongodb*. Which is also redundant now since all of our tests start with test_mongodb. So test_mongodb* should work

Okay, got it. Will make this change in latest commit.

luandy64 · 2023-07-25T17:58:08Z

.circleci/config.yml

+            selected_tests=$(circleci tests glob "tests/*{test_mongodb}*")
+            echo -e "Tests running in this batch:\n$selected_tests\n"
+            echo $selected_tests > ./tests-to-run
+            if [ -s ./tests-to-run ]; then
+              for test_file in $(cat ./tests-to-run)
+              do
+                run-test --tap=tap-mongodb $test_file
+              done
+            else
+              echo "ERROR - File not found or file is empty!" >&2
+              exit 5
+            fi


If this is to enable parallelism, I think it's incomplete.

From the latest test run, you can see that Tests running in this batch gets printed once with all of the tests. Also in CircleCI, the workflow doesn't show more than one node running the tests

@luandy64 this was not to enable parallelism as the builds were running in batch already.
This change was done to include MongoDB version 6.0 for tests & remove MongoDB version 4.2 as it has reached End of Life.
But, we can execute them in parallel & I can make this change, if it's a good idea.

1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values.

- sort the dictionary before assertion

…oddb v6.0

- improve integration testlogging

…-mongodb into TDL-23471-circle-ci-config

…ng in CCi.

* - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * [Tdl 23471] Fix circle ci config (#101) * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> * Tdl 23609 fix empty projection (#102) * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com>

* Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Tdl 23543 fix uuid transformation (#100) * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * [Tdl 23471] Fix circle ci config (#101) * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> * Tdl 23609 fix empty projection (#102) * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> * renamed build_mongo_4_4 to build in config.yml * Renamed build_mongo_4_4 to build in config.yml --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

* Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Major version bump 3.0.0 * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * Tdl 23543 fix uuid transformation (#100) * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * fix typo * remove f-string from logger statements * minor code cleanup * [Tdl 23471] Fix circle ci config (#101) * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * Added CircleCI config for mongodb 6.0. * Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0 * Fixed CCI config and added all tests for mongodb 4.4 * Removed mongodb 4.2 testing from CCi as it has reached end of life. * Fixed redundant statement to include all integration tests. * Removed glob & for loop statements as all integration tests are running in CCi. --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> * Tdl 23609 fix empty projection (#102) * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * Force {} projection to be None * Changes: 1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values. * - set maxDiff to None to reset dict comparison limit - sort the dictionary before assertion * fix uuid transformation * upgrade pymongo version in cci configuration * fix cci configuration * upgrade pip and setuptools * update cci config to run on latest python version * fix linting issues * fix uuid bookmarking * fix integration tests * remove debugger import * fix integration test compatibility issues with pymongo v4.3+ and mongoddb v6.0 * - fix transaction life time limit - improve integration testlogging * projection test change * - fix the emtpy projection issue - fix the integration tests * minor integration test fixes * remove redundant comments * fix typo * remove f-string from logger statements * Force {} projection to be None --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: shantanu73 <shantanudhiman25@gmail.com> * Removed CCi changes from CHANGELOG.md * renamed build_mongo_4_4 to build in config.yml * Renamed build_mongo_4_4 to build in config.yml * Major version bump 3.0.0 * Removed CCi changes from CHANGELOG.md --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

Added CircleCI config for mongodb 6.0.

0bd9d25

shantanu73 changed the base branch from master to TDL-23543-fix-uuid-transformation July 20, 2023 13:02

Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0

8d04633

shantanu73 changed the base branch from TDL-23543-fix-uuid-transformation to master July 20, 2023 14:05

shantanu73 changed the base branch from master to TDL-23543-fix-uuid-transformation July 20, 2023 14:05

shantanu73 added 2 commits July 20, 2023 17:47

Fixed CCI config and added all tests for mongodb 4.4

c9588e5

Removed mongodb 4.2 testing from CCi as it has reached end of life.

fbad252

shantanu73 requested a review from luandy64 July 24, 2023 08:14

luandy64 reviewed Jul 25, 2023

View reviewed changes

shantanu73 and others added 20 commits July 26, 2023 11:25

Fixed redundant statement to include all integration tests.

e66cf57

Changes:

03f068c

1) Upgrade pymongo to 4.4.0 2) Added condition to handle DatetimeMS binary date values.

- set maxDiff to None to reset dict comparison limit

54c3838

- sort the dictionary before assertion

fix uuid transformation

58b76df

upgrade pymongo version in cci configuration

f98de13

fix cci configuration

648bfd5

upgrade pip and setuptools

cc404d8

update cci config to run on latest python version

f89e869

fix linting issues

5324f3a

fix uuid bookmarking

051d287

fix integration tests

93655ea

remove debugger import

415990a

fix integration test compatibility issues with pymongo v4.3+ and mong…

3cdab90

…oddb v6.0

- fix transaction life time limit

ada41ca

- improve integration testlogging

Added CircleCI config for mongodb 6.0.

f6944a4

Added Ciall testsoin CCI for ngodb 6.5.0 & 6.0

0fb8ecd

Fixed CCI config and added all tests for mongodb 4.4

8c86384

Removed mongodb 4.2 testing from CCi as it has reached end of life.

4645e3d

Fixed redundant statement to include all integration tests.

b9fc7c5

Merge branch 'TDL-23471-circle-ci-config' of github.com:singer-io/tap…

c69a57b

…-mongodb into TDL-23471-circle-ci-config

shantanu73 changed the base branch from TDL-23543-fix-uuid-transformation to master July 26, 2023 11:50

shantanu73 changed the base branch from master to TDL-23543-fix-uuid-transformation July 26, 2023 11:50

Removed glob & for loop statements as all integration tests are runni…

5c0f2f3

…ng in CCi.

luandy64 approved these changes Jul 26, 2023

View reviewed changes

shantanu73 merged commit 72bbf4c into TDL-23543-fix-uuid-transformation Jul 27, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tdl 23471] Fix circle ci config #101

[Tdl 23471] Fix circle ci config #101

shantanu73 commented Jul 20, 2023 •

edited

Loading

luandy64 Jul 25, 2023

shantanu73 Jul 26, 2023

luandy64 Jul 25, 2023

shantanu73 Jul 26, 2023

[Tdl 23471] Fix circle ci config #101

[Tdl 23471] Fix circle ci config #101

Conversation

shantanu73 commented Jul 20, 2023 • edited Loading

Description of change

QA steps

Risks

Rollback steps

luandy64 Jul 25, 2023

Choose a reason for hiding this comment

shantanu73 Jul 26, 2023

Choose a reason for hiding this comment

luandy64 Jul 25, 2023

Choose a reason for hiding this comment

shantanu73 Jul 26, 2023

Choose a reason for hiding this comment

shantanu73 commented Jul 20, 2023 •

edited

Loading