New Source: Airbyte-native Mailgun

[sherifnada]

#5904 introduced a singer-based version of mailgun contributed by a community member. We want to support this connector but due to licensing and technical issues with Singer we need to move it to a native connector.

[yevhenii-ldv]

Integration Vetting

Webhook-based? (no/partially/yes)
Supproted but not required

Available authentication modes (API key/Oauth/other)
HTTP Basic Auth (API key)

Creating an account
Free account is registerred: integration-test@airbyte.io
Paid plans are available.
Here’s the plans comparison: https://app.mailgun.com/app/billing/plans
From $35 to $1250 per month.
Key features available only for paid plans:
Receiving messages (Routes)

How to populate the account with data?
Using API. Required initial settings in UI are done.

Available streams for sync
API reference: https://documentation.mailgun.com/en/latest/api_reference.html

Bounces https://documentation.mailgun.com/en/latest/api-suppressions.html#bounces
Complaints https://documentation.mailgun.com/en/latest/api-suppressions.html#complaints
Domains https://documentation.mailgun.com/en/latest/api-domains.html
Events https://documentation.mailgun.com/en/latest/api-events.html
IPs https://documentation.mailgun.com/en/latest/api-ips.html
IP pools https://documentation.mailgun.com/en/latest/api-ip-pools.html
Mailing lists https://documentation.mailgun.com/en/latest/api-mailinglists.html
Messages https://documentation.mailgun.com/en/latest/api-sending.html
Results https://documentation.mailgun.com/en/latest/api-inbox-placement.html#seed-results
Routes https://documentation.mailgun.com/en/latest/api-routes.html
Seed lists https://documentation.mailgun.com/en/latest/api-inbox-placement.html#generate-a-seed-list
Stats https://documentation.mailgun.com/en/latest/api-stats.html
Tags https://documentation.mailgun.com/en/latest/api-tags.html
Templates https://documentation.mailgun.com/en/latest/api-templates.html
Unsubscribes https://documentation.mailgun.com/en/latest/api-suppressions.html#unsubscribes
Validate https://documentation.mailgun.com/en/latest/api-email-validation.html#
Webhooks https://documentation.mailgun.com/en/latest/api-webhooks.html
Whitelists https://documentation.mailgun.com/en/latest/api-suppressions.html#whitelists

Do any streams support incremental sync?
Partly, i.e. it's possible to use datetime data for messages.

[sergei-solonitcyn]

Product Requirements & features

1. What is the “ideal end state” for the features offered by this connector?

• Mailgun REST entities (subentities) map to Airbyte streams
• Only API key is required to make requests

2. What is the release roadmap for this connector?

• Domains and Messages endpoints seem to be required for very minimal functionality
• All other endpoints available in the free plan would provide advanced benefits
• These endpoints require a paid plan and could be implemented afterwards:
  • Inbox subentities (Results, Seed lists)
  • IP pools
  • Routes
  • Validate

3. Check if and how Fivetran or Stitch Data support this connector

• Fivetran doesn’t support Mailgun and doesn’t plan to do it. Related request still has only one vote for it: https://support.fivetran.com/hc/en-us/community/posts/4408445558807-New-Connector-Mailgun
• Integration solution by Stitchdata also isn’t detected

[sergei-solonitcyn]

Implementation Scoping

Commonalities across the API endpoints

• Pagination strategy

  According to the docs, some endpoints (ex.: tags) have a <limit> field with different default and max values for different endpoints. The responses provide first, last, previous and next page urls (or just previous and next).
  Some endpoints (ex.: domains/credentials) provide skip and limit instead.

• Authentication strategy

  HTTP Basic Auth with Private API key is used for every request, so it could be used as a config data for the connector

• Populate the sandbox account with data

  It’s possible to create a script which would automatically generate some data. It would use the API functionality itself, but the UI doesn’t provide the whole functionality

• Handling rate limits

  In case of exceeding rate limits (300 requests per minute) the endpoints return a rate limit response:

{
        "retry-seconds": 60,
}

Which should be handled in the obvious way

[sherifnada]

@sergei-solonitcyn thanks for providing the context! the initial two streams you proposed sound good for a start. Let's move ahead with that scope! Maybe also let's add events, since it's a useful stream that implements useful incremental functionality

[sergei-solonitcyn]

Agree, Events is defenitely good one to be in the MVP.
Getting to work.