extract out the connector changelog modification out of the bump_version code #34586

stephane-airbyte · 2024-01-27T21:34:36Z

We want to move the changelog mutation out of the connectors command, so we can reuse it for java CDK.
all connectors were tested, and the only ones that were failing are fixed in #36039

vercel · 2024-01-27T21:34:37Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Mar 14, 2024 0:11am

stephane-airbyte · 2024-01-27T21:34:48Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @stephane-airbyte and the rest of your teammates on Graphite

alafanechere

The overall logic looks perfect to me. I made a couple of suggestion to boost reusability and make it a bit more refactor proof.

alafanechere · 2024-02-01T17:12:04Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+    def __init__(self, changelog_file_path: Path) -> None:
+        self.changelog_file_path = changelog_file_path
+        if not changelog_file_path.exists():
+            raise Exception(f"File {changelog_file_path} does not exist")
+        if not changelog_file_path.is_file():
+            raise Exception(f"File {changelog_file_path} is not a regular file")
+        self.original_markdown_lines = changelog_file_path.read_text().splitlines()


Can we pass the original_markdown_lines as a constructor parameter and avoid path checks logic there?
I'd love to avoid direct filesystem access if possible.
I'd prefer the changelog read to happen from a dagger directory object. This will help us make this Changelog agnostic of the underlying filesystem / path: a dagger directory could come from a container, a remote git repo or the host fs.

Feel free to not change how the documentation file path is fetched, but please change the constructor to take the raw markdown string as an input instead of the file path

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

alafanechere · 2024-02-01T17:23:15Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+        changelog_entry_re = (
+            "^\\| *(?P<version>[0-9]+\\.[0-9+]+\\.[0-9]+?) *\\| *"
+            + "(?P<day>[0-9]{4}-[0-9]{2}-[0-9]{2}) *\\| *"
+            + "\\[?(?P<pr_number1>[0-9]*)\\]? ?\\(https://github.com/airbytehq/airbyte/pull/(?P<pr_number2>[0-9]*)\\) *\\| *"


Can you add a github_repo_url parameter to the constructor which defaults to a constant built from this one?
It'll make it less coupled to our repo if we allow airbyte-ci to run on different repo anytime

alafanechere · 2024-02-01T17:24:36Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+        )
+        return "\n".join(new_lines) + "\n"
+
+    def __parse_markdown(self) -> None:


Please add a docstring describing the parsing logic?

alafanechere · 2024-02-01T17:26:07Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+                changelog_header_line_index = line_index
+                break
+        if changelog_header_line_index == -1:
+            raise Exception("Could not find the changelog section table in the documentation file.")


Can you create a custom ChangelogParsing exception? Exception is too broad 😀

alafanechere · 2024-02-01T17:30:15Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+            self.old_entries.add(changelog_entry)
+
+
+class ChangelogEntry:


Can we use a frozen dataclass there? It'll make the class leaner and make it explicitely a stateless object

I have no idea what a "frozen dataclass" is... I guess I have some reading to do

You can consider Dataclass to be models / POJO. They're could frozen when they're immutable.

alafanechere · 2024-02-01T17:30:51Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+        self.pr_number = int(pr_number)
+        self.comment = comment
+
+    def to_str(self) -> str:


Suggested change

def to_str(self) -> str:

def to_markdown(self) -> str:

or

Suggested change

def to_str(self) -> str:

def __str__(self) -> str:

if you'd like to call str(changelog_entry)

alafanechere · 2024-02-01T17:35:00Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+        )
+        return "\n".join(new_lines) + "\n"
+
+    def __parse_markdown(self) -> None:


Can we make this a public static or class method which would return a list of changelog entry?
I'd find it more elegant to call it this way in the contrusctor:
self.old_entries = self.parse_markdown(original_markdown_line)

Making it public and a static/class method can simplify testing and reuse in other contexts.

the parse_markdown does a lot more than just returning the old entries. It also sets the start and end lines of the changelog. I guess we could redo that in the to_markdown

alafanechere · 2024-02-01T17:35:41Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

This is exactly the approach I suggested before, thank you so much!

airbyte-ci/connectors/pipelines/tests/test_changelog.py

stephane-airbyte · 2024-02-02T05:04:13Z

@alafanechere I believe I addressed all your comments.

alafanechere

@stephane-airbyte the code looks good to me now! Thank you for the changes. I added a minor suggestion.
I manually tried it on a couple of python connectors and realized:

The error reporting was not working. I fixed it in 1c91364
The exception thrown is The changelog table in the documentation file is missing the header delimiter.

Can you have a look to make sure the majority of our python connectors can be bumped with this command?
I tried:
airbyte-ci connectors --name=source-stripe bump_version minor 123 "new stuff"

alafanechere · 2024-02-02T13:30:39Z

airbyte-ci/connectors/pipelines/pipelines/helpers/changelog.py

+        return self.__str__().__hash__()
+
+
+def parse_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:


Suggested change

def parse_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:

def parse_changelog_in_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:

Can you please add a docstring?

…ion code

alafanechere

I fixed type checks which failed the CI.
I tried the command on python connectors and it works as expected 👌
Thanks Stephane for this improvement!

stephane-airbyte mentioned this pull request Jan 27, 2024

add connector commands to add entries to changelog #34371

Closed

vercel bot deployed to Preview January 27, 2024 21:46 View deployment

stephane-airbyte force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch 9 times, most recently from 0ea0a74 to f3301f8 Compare January 31, 2024 19:50

stephane-airbyte marked this pull request as ready for review January 31, 2024 20:11

stephane-airbyte requested review from a team and alafanechere January 31, 2024 20:11

alafanechere requested changes Feb 1, 2024

View reviewed changes

stephane-airbyte force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch 5 times, most recently from 7b3473a to 9338a2e Compare February 2, 2024 05:03

stephane-airbyte requested a review from alafanechere February 2, 2024 05:03

alafanechere requested changes Feb 2, 2024

View reviewed changes

stephane-airbyte force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch from 1c91364 to b12bbd7 Compare March 13, 2024 17:53

stephane-airbyte requested a review from a team as a code owner March 13, 2024 17:53

stephane-airbyte mentioned this pull request Mar 13, 2024

add connectors command to add entries to the changelog #36028

Closed

stephane-airbyte force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch 2 times, most recently from 502a276 to 4b43fa9 Compare March 14, 2024 01:34

stephane-airbyte force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch from 4b43fa9 to 723d101 Compare March 14, 2024 01:40

extract out the connector changelog modification out of the bump_vers…

b49c1d1

…ion code

alafanechere force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch from 723d101 to 848e8a7 Compare March 14, 2024 11:20

alafanechere approved these changes Mar 14, 2024

View reviewed changes

fix type checks

51659f3

alafanechere force-pushed the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch from 848e8a7 to 51659f3 Compare March 14, 2024 12:11

stephane-airbyte merged commit 3fae982 into master Mar 14, 2024
31 checks passed

stephane-airbyte deleted the stephane/01-27-extract_out_the_connector_changelog_modification_out_of_the_bump_version_code branch March 14, 2024 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extract out the connector changelog modification out of the bump_version code #34586

extract out the connector changelog modification out of the bump_version code #34586

stephane-airbyte commented Jan 27, 2024 •

edited

vercel bot commented Jan 27, 2024 •

edited

stephane-airbyte commented Jan 27, 2024 •

edited

alafanechere left a comment

alafanechere Feb 1, 2024

stephane-airbyte Mar 14, 2024

alafanechere Feb 1, 2024

stephane-airbyte Mar 14, 2024

alafanechere Feb 1, 2024

alafanechere Feb 1, 2024

stephane-airbyte Mar 14, 2024

alafanechere Feb 1, 2024

stephane-airbyte Feb 1, 2024

alafanechere Feb 2, 2024

stephane-airbyte Mar 14, 2024

alafanechere Feb 1, 2024

stephane-airbyte Mar 14, 2024

alafanechere Feb 1, 2024

stephane-airbyte Feb 1, 2024

alafanechere Feb 1, 2024

stephane-airbyte commented Feb 2, 2024

alafanechere left a comment

alafanechere Feb 2, 2024

alafanechere Feb 2, 2024

alafanechere left a comment

		return self.__str__().__hash__()


		def parse_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:

	def parse_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:
	def parse_changelog_in_markdown(markdown_lines: list[str], github_repo: str) -> [int, set[ChangelogEntry]]:

extract out the connector changelog modification out of the bump_version code #34586

extract out the connector changelog modification out of the bump_version code #34586

Conversation

stephane-airbyte commented Jan 27, 2024 • edited

vercel bot commented Jan 27, 2024 • edited

stephane-airbyte commented Jan 27, 2024 • edited

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephane-airbyte commented Feb 2, 2024

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

stephane-airbyte commented Jan 27, 2024 •

edited

vercel bot commented Jan 27, 2024 •

edited

stephane-airbyte commented Jan 27, 2024 •

edited