🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type #8161

andriikorotkov · 2021-11-21T15:52:03Z

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…6544-mongo-performance-issues

…6544-mongo-performance-issues � Conflicts: � airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e.json � airbyte-config/init/src/main/resources/seed/source_definitions.yaml � airbyte-integrations/connectors/source-mongodb-strict-encrypt/Dockerfile � airbyte-integrations/connectors/source-mongodb-v2/Dockerfile � docs/integrations/sources/mongodb-v2.md

…6544-mongo-performance-issues

andriikorotkov · 2021-11-22T15:19:51Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1490850430
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1490850430
🐛 https://gradle.com/s/thybrow25oogc

andriikorotkov · 2021-11-22T15:20:44Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1490854178
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1490854178
No Python unittests run

andriikorotkov · 2021-11-22T15:36:07Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1490919244
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1490919244
No Python unittests run

alexandr-shegeda

LGTM. we can proceed with publishing and merge without airbyte review

andriikorotkov · 2021-11-23T11:05:26Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1494441804
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1494441804
No Python unittests run

andriikorotkov · 2021-11-23T11:05:42Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1494442698
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1494442698
No Python unittests run

andriikorotkov · 2021-11-23T11:15:40Z

/publish connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1494479769
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1494479769

andriikorotkov · 2021-11-23T11:16:08Z

/publish connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1494481235
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1494481235

…amp type (airbytehq#8161) * updated source-mongodb-v2 performance * updated code style * fixed remarks * fixed remarks * fixed remarks * updated strict encrypt source mongodb version * updated source mongodb work with empty collections * updated source mongodb timestamp cursor * updated mongodb source perfomance * fix code style * fix code style * updated tests and documentation * updated tests and documentation * updated tests and documentation * added vudangngoc changes * updated code style * updated code style

andriikorotkov added 11 commits November 15, 2021 16:47

updated source-mongodb-v2 performance

f77238f

updated code style

6ac8c93

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

6e3f202

…6544-mongo-performance-issues

fixed remarks

9ff2bfd

fixed remarks

4eb9d09

fixed remarks

451fe71

updated strict encrypt source mongodb version

ae3c06e

updated source mongodb work with empty collections

a1d23ff

updated source mongodb timestamp cursor

799c877

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

a5d444c

…6544-mongo-performance-issues

andriikorotkov requested a review from alexandr-shegeda November 21, 2021 15:52

andriikorotkov temporarily deployed to more-secrets November 21, 2021 15:53 Inactive

updated mongodb source perfomance

e83f8d3

andriikorotkov temporarily deployed to more-secrets November 21, 2021 20:42 Inactive

fix code style

9a8dce4

andriikorotkov temporarily deployed to more-secrets November 22, 2021 10:49 Inactive

fix code style

18fd7b7

andriikorotkov temporarily deployed to more-secrets November 22, 2021 11:08 Inactive

updated tests and documentation

70ccff2

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 22, 2021

updated tests and documentation

868d7e9

andriikorotkov changed the title ~~Akorotkov/6544 mongo performance issues~~ 🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type Nov 22, 2021

andriikorotkov temporarily deployed to more-secrets November 22, 2021 15:22 Inactive

jrhizor temporarily deployed to more-secrets November 22, 2021 15:22 Inactive

updated tests and documentation

a3c6284

andriikorotkov temporarily deployed to more-secrets November 22, 2021 15:39 Inactive

alexandr-shegeda approved these changes Nov 22, 2021

View reviewed changes

andriikorotkov mentioned this pull request Nov 22, 2021

Fix NPE when field list out of sync #8157

Closed

40 tasks

added vudangngoc changes

d6399b3

andriikorotkov temporarily deployed to more-secrets November 23, 2021 10:44 Inactive

updated code style

d427834

andriikorotkov temporarily deployed to more-secrets November 23, 2021 10:56 Inactive

andriikorotkov marked this pull request as ready for review November 23, 2021 11:00

jrhizor temporarily deployed to more-secrets November 23, 2021 11:07 Inactive

jrhizor temporarily deployed to more-secrets November 23, 2021 11:17 Inactive

jrhizor temporarily deployed to more-secrets November 23, 2021 11:18 Inactive

updated code style

7a8c313

andriikorotkov temporarily deployed to more-secrets November 23, 2021 11:26 Inactive

andriikorotkov merged commit 6bf9eba into master Nov 23, 2021

andriikorotkov deleted the akorotkov/6544-mongo-performance-issues branch November 23, 2021 11:56

This was linked to issues Nov 25, 2021

Source: MongoDB - Timestamp microseconds missing in Cursor - Incremental change capture #7556

Closed

Performance issues in new Mongo Source Connector #6544

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type #8161

🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type #8161

andriikorotkov commented Nov 21, 2021

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type #8161

🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type #8161

Conversation

andriikorotkov commented Nov 21, 2021

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

andriikorotkov commented Nov 22, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 22, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 22, 2021 • edited by github-actions bot Loading

alexandr-shegeda left a comment

Choose a reason for hiding this comment

andriikorotkov commented Nov 23, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 23, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 23, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 23, 2021 • edited by github-actions bot Loading

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 22, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Nov 23, 2021 •

edited by github-actions bot

Loading