Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 New Source: Close.com #5366

Merged
merged 42 commits into from
Sep 13, 2021
Merged

🎉 New Source: Close.com #5366

merged 42 commits into from
Sep 13, 2021

Conversation

annalvova05
Copy link
Contributor

@annalvova05 annalvova05 commented Aug 12, 2021 •
edited
Loading

What

Resolving the issue:
#4231 - New Source: Close.com

How

Using CDK and custom methods implemented the connector logic for Close.com as a source for Airbyte.

Recommended reading order

  1. source.py

Pre-merge Checklist

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 12, 2021
@annalvova05
Copy link
Contributor Author

annalvova05 commented Aug 12, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1124151850
❌ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1124151850

@annalvova05 annalvova05 linked an issue Aug 12, 2021 that may be closed by this pull request
New Source: Close.com #4231
Closed
@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 14:00 Inactive
@annalvova05
Copy link
Contributor Author

annalvova05 commented Aug 12, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1124375615
❌ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1124375615

@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 15:06 Inactive
bazarnov
bazarnov requested changes Aug 12, 2021
View reviewed changes
Copy link
Collaborator

@bazarnov bazarnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done, but there is room to improve right away, please check the comments and make the changes accordingly.

airbyte-integrations/connectors/source-close-com/acceptance-test-config.yml Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/acceptance-test-config.yml Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/sample_files/config.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/sample_files/configured_catalog.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/sample_files/state.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
@annalvova05
Copy link
Contributor Author

annalvova05 commented Aug 12, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1125169817
❌ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1125169817

@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 19:28 Inactive
midavadim
midavadim suggested changes Aug 16, 2021
View reviewed changes
airbyte-integrations/connectors/source-close-com/source_close_com/spec.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Show resolved Hide resolved
airbyte-integrations/connectors/source-close-com/source_close_com/source.py Show resolved Hide resolved
@annalvova05
update after review
8cb55a6
@annalvova05
Merge remote-tracking branch 'origin/master' into alvova/4231-source-…
116bea8 
…close-com

# Conflicts:
#	airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@annalvova05
Copy link
Contributor Author

annalvova05 commented Aug 17, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1140156409
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1140156409

@jrhizor jrhizor temporarily deployed to more-secrets August 17, 2021 17:08 Inactive
@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 8, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1213253285
❌ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1213253285

@jrhizor jrhizor temporarily deployed to more-secrets September 8, 2021 12:13 Inactive
@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 8, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1213444776
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1213444776

@jrhizor jrhizor temporarily deployed to more-secrets September 8, 2021 13:12 Inactive
Phlair
Phlair suggested changes Sep 9, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updates @annalvova05! Great stuff.

One thing I'm still not 100% confident on is the potential for skipping data with the combination of date filter and _skip/_limit. This could only happen if the state is being updated during the sync, which is avoidable by removing state_checkpoint_interval so we only save state at end of each sync. (See how state is getting saved via abstract_source here)
Does that make sense or am I missing something here?

airbyte-integrations/connectors/source-close-com/source_close_com/source.py Outdated Show resolved Hide resolved
@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 9, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1217535901
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1217535901

@jrhizor jrhizor temporarily deployed to more-secrets September 9, 2021 13:42 Inactive
@annalvova05
Copy link
Contributor Author

@Phlair thanks, understand the problem! I think is a good idea to remove state_checkpoint_interval due to date filters.
Updated

Phlair
Phlair approved these changes Sep 10, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, thanks @annalvova05 👍

@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1221597571
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1221597571

@jrhizor jrhizor temporarily deployed to more-secrets September 10, 2021 14:28 Inactive
@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 13, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1229934579
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1229934579

@jrhizor jrhizor temporarily deployed to more-secrets September 13, 2021 14:23 Inactive
@annalvova05
Copy link
Contributor Author

annalvova05 commented Sep 13, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1229967759
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/1229967759

@jrhizor jrhizor temporarily deployed to more-secrets September 13, 2021 14:32 Inactive
@annalvova05 annalvova05 merged commit 03e1e08 into master Sep 13, 2021
@annalvova05 annalvova05 deleted the alvova/4231-source-close-com branch September 13, 2021 14:44
@jrhizor jrhizor mentioned this pull request Sep 16, 2021
Merged
@zkid18
Copy link
Contributor

zkid18 commented Feb 17, 2022

@annalvova05 any reason why custom fields were not included? I'm thinking of a contribution, so want to know if there is anything need to be aware of

@annalvova05
Copy link
Contributor Author

@zkid18 It's a bug. Issue for this has already been created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@midavadim midavadim midavadim approved these changes

@bazarnov bazarnov bazarnov approved these changes

@Phlair Phlair Phlair approved these changes

Assignees

@annalvova05 annalvova05

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/close-com connectors/sources-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

New Source: Close.com
7 participants
@annalvova05 @Phlair @zkid18 @midavadim @bazarnov @jrhizor @igrankova