✨Source MailGun: Migrate Python CDK to Low-code CDK

[btkcodedev]

What

Migrating Source Mailgun to Low-Code CDK
Closes #29120

How

Developed using (Configuration Based Source) low-code CDK

Recommended reading order

1. spec.yaml
2. manifest.yaml
3. schemas/*

Tests

Integration & Acceptance

Full Test Results

🚨 User Impact 🚨

• No breaking changes, migration to low-code

Migrate Connector: Source Mailgun

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[btkcodedev]

Test results: Passed state:

[btkcodedev]

Waiting for #28471 for merging to update start_date

[marcosmarxm]

Waiting for #28471 for merging to update start_date

Go ahead and fix the start date directly here.

[btkcodedev]

@marcosmarxm It's Hitting with backward compatibility test errors, any suggestions?

[marcosmarxm]

Left some comments, the connector is missing pagination.

[marcosmarxm]

Why ar eyou changing this?

[marcosmarxm]

Add a date here

[marcosmarxm]

Please add pagination as it was implemented before

airbyte/airbyte-integrations/connectors/source-mailgun/source_mailgun/source.py

Lines 42 to 49 in b78ad4e

	def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, HttpUrl]]:
	"""
	:param response: the most recent response from the API
	:return If there is another page in the result, a mapping (e.g: dict) containing information needed to query the next page in the response.
	If there are no more pages in the result, return None.
	"""
	next_page: Optional[HttpUrl] = response.json().get("paging", {}).get("next")
	return {"url": next_page} if next_page and self._pre_parse_response(response) else None

[marcosmarxm]

EOF

[btkcodedev]

All comments resolved
Test result passed state

[btkcodedev]

[sajarin]

@btkcodedev getting an error with the check command with the events stream:

{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an error trying to connect to stream events. Error: \n Traceback (most recent call last):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py\", line 42, in check_connection\n stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py\", line 36, in check_availability\n stream_slice = get_first_stream_slice(stream)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py\", line 21, in get_first_stream_slice\n stream.stream_slices(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 126, in stream_slices\n return self.retriever.stream_slices()\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 348, in stream_slices\n return self.stream_slicer.stream_slices()\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py\", line 141, in stream_slices\n start_datetime = self._calculate_earliest_possible_value(self._select_best_end_datetime())\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py\", line 146, in _calculate_earliest_possible_value\n earliest_possible_start_datetime = min(self.start_datetime.get_datetime(self.config), end_datetime)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/min_max_datetime.py\", line 58, in get_datetime\n time = self._parser.parse(str(self.datetime.eval(config, **additional_parameters)), datetime_format)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/datetime_parser.py\", line 29, in parse\n parsed_datetime = datetime.datetime.strptime(str(date), format)\n File \"/usr/local/lib/python3.9/_strptime.py\", line 568, in _strptime_datetime\n tt, fraction, gmtoff_fraction = _strptime(data_string, format)\n File \"/usr/local/lib/python3.9/_strptime.py\", line 349, in _strptime\n raise ValueError(\"time data %r does not match format %r\" %\nValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'\n"}}

{"type": "LOG", "log": {"level": "ERROR", "message": "Check failed"}}

{"type": "CONNECTION_STATUS", "connectionStatus": {"status": "FAILED", "message": "\"Unable to connect to stream events - time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'\""}}

[btkcodedev]

@sajarin
Updated the start date for solving #27385,

Please update the secrets config to match the updated format (start_date: %Y-%m-%dT%H:%M:%SZ)
Ping via Slack (btkcodedev) for getting my creds if needed

[sajarin]

@btkcodedev thanks for the update, config validation is now passing but no records are being fetched. I pinged you on slack for credentials so we can get this moving.

changes addressed

[sajarin]

Thanks for the contribution @btkcodedev!

[padala]

@btkcodedev @sajarin Thanks for the contribution. I'm still facing the issue with sync resulting in 0 records when I tried this today on Airbyte Cloud. Is this already resolved?

@btkcodedev thanks for the update, config validation is now passing but no records are being fetched. I pinged you on slack for credentials so we can get this moving.

[sajarin]

@padala can you make a new issue and tag me? Perhaps an issue with the record selector

[padala]

@sajarin I've created a new issues here - #30105