C4GT 2023: Write an Airbyte connector for Avni

[fatchat]

Intro

Airbyte is an open-source ELT tool with a large ecosystem of connectors to various data sources and destinations. They encourage the development of connectors by third-parties, and their platform is designed to work with connectors adhering to their specification.

Avni is a data-collection platform used widely in the development sector. They allow the creation of complex forms and workflows for field workers, with features like multi-language support and offline-mode.

API Guide for AVNI

https://avni.readme.io/docs/api-guide

The Task

We would like to build a source connector for Avni. Doing this will allow Avni's NGO users to leverage Airbyte for their data management needs.

Airbyte connectors can be written in Java or Python. The Tech4Dev team has experience in building connectors for other development sector data-collection tools, for example:

• CommCare
• SurveyCTO

We will guide you through the design, development and eventual submission to Airbyte.

You will learn how to contribute to an open-source project, including

• following their coding guidelines
• writing documentation
• writing unit and integration tests for CI/CD
• communicating with their team on Slack
• submitting the PR

If you want to simultaneously contribute to an open-source project and to the development sector, this is a great project for you.

Features to be implemented

• An Airbyte source connector for Avni

Learning Path

Complexity	Moderate
Risk	Low
Skills required	Java / Python
Name of Mentor	@siddhant3030
Project Size

Project setup

Read this and clone this repository

Acceptance Criteria

• Airbyte should ingest sample data
• Documentation and tests as required by Airbyte
• Note that Airbyte may not merge the PR within the timeline of the project, that is out of our control

Milestones

Understand the Platform for Avni and create a sandbox account:

In this milestone, you need to familiarize yourself with the Avni platform. Start by creating a sandbox account on Avni. A sandbox account provides a safe environment for development and testing without affecting production data.

Explore the APIs of Avni :

Once you have your sandbox account, explore the Avni APIs documentation. Understand the available endpoints, request methods (such as GET, POST, PUT, DELETE), and request/response formats (such as JSON, XML).

Implement the basic read of the connector:

Choose a specific connector you want to work with and implement the basic read functionality. This means retrieving data from the connector using the Avni API. This could involve making a GET request to the appropriate endpoint with the required parameters, such as specifying the data source and any filters. Process the API response and extract the relevant data that you need.

Check the API success while making a connection and throw some errors if the credentials are wrong:

When making API requests to Avni, it's essential to handle potential errors effectively. Implement a mechanism to check the success of the API connection before proceeding further. This can involve checking the response status code, such as a 200 OK indicating a successful connection. Additionally, if the authentication credentials are incorrect, handle the error gracefully by providing appropriate feedback or error messages to the user.

Pull data from a particular date:

Extend your read functionality to pull data from a specific date or date range. You may need to modify your API request parameters to include the desired date or date range as filters. This allows you to retrieve only the data that has been updated or added since the specified date, improving efficiency when synchronizing data.

Implement incremental sync:

Incremental sync is the process of synchronizing only the changes or updates since the last synchronization instead of retrieving all the data again. Implement this functionality by keeping track of the last synchronization timestamp or a synchronization token. Use this information to fetch only the modified or new data from Avni since the last synchronization. Update your local system accordingly to reflect the changes.

Write tests

Create test cases to ensure the functionality of your connector. Use Airbyte's provided testing utilities and examples to guide you.

Document your connector

Document your connector to provide clear instructions on how to use it. This includes documenting the connector's configuration options, any specific considerations or limitations, and examples of usage. Follow Airbyte's documentation guidelines to ensure consistency.

How to contribute

Familiarize yourself with the project

Start by exploring the Airbyte project on GitHub. Visit the project repository and read through the documentation, including the README file, contributing guidelines, and code of conduct. Understand the project's goals, architecture, and the current state of development.

Set up the development environment

Follow the instructions provided in the Airbyte repository to set up the development environment. This typically involves installing the necessary dependencies, setting up a local development instance, and running tests to ensure everything is working correctly.

Fork the repository

Fork the Airbyte repository on GitHub to create your own copy of the project. This will allow you to make changes and submit pull requests.

Create a new branch

Create a new branch in your forked repository to work on your contribution. It's recommended to create a branch with a descriptive name that reflects the nature of your changes.

Make the necessary changes

Write code, fix bugs, or make the desired improvements in your branch. Follow the project's coding standards and conventions. Make sure to write clear and concise commit messages for each change.

Test your changes

Ensure that your changes work as expected by running tests provided by the project. This helps maintain the quality and stability of the codebase.

Submit a pull request

Once you are confident with your changes, submit a pull request (PR) to the main Airbyte repository. Provide a detailed description of your changes, explaining the problem you addressed or the feature you added. Include any relevant documentation updates and tests.

Collaborate and iterate

Engage in discussions with the Airbyte community and maintainers regarding your contribution. Address any feedback or code review comments promptly. Iterate on your changes as necessary until they are ready to be merged into the main codebase.

[Shruti3004]

Hey @fatchat can you please also add Project size and milestones

[fatchat]

@siddhant3030 could you please resolve Shruti's comment

[siddhant3030]

Hey @fatchat can you please also add Project size and milestones

Hey. The project size is Large and I have added a few milestones which can help you focus. Let me know if you want to discuss more

[Siddhant2021]

hi @siddhant3030 , i am new to open source and a bit lost after the project setup part. Can you brief a little on what we have to do (that's the acceptance criteria part i don't understand much).

[siddhant3030]

@Shruti3004 @Siddhant2021 are you guys on discord? We can help you there and connect with you guys if you need help. here's the invite link https://discord.gg/5etKnjPx . Do join us

[Shanzal04]

Hey @siddhant3030 , can u please elaborate the milestone a bit more.

[siddhant3030]

@Shanzal04 I have added a few details. Please look into that and let me know if you didn't understand something.

[SigireddyBalasai]

I am looking forward to contribute

[FirePing32]

Hey @Shruti3004 @fatchat
This seems like an interesting project. I have worked on something similar before and am eager to contribute to this project. Looking forward to submitting a proposal soon 🚀

[Darshan2003]

@Shruti3004 @Siddhant2021 are you guys on discord? We can help you there and connect with you guys if you need help. here's the invite link https://discord.gg/5etKnjPx . Do join us

Hey @siddhant3030, the invite link seems to be expired can you share the new link?