upgrade dbt_utils dependency

[mateuszklimek]

dbt_utils is currently on version 0.9.2 but fivetran_utils supports only version < 0.9.0:

packages:

  - package: dbt-labs/dbt_utils
    version: [">=0.8.0", "<0.9.0"]

We at re_data use fiveran_utils and already got a couple of requests to upgrade to a newer version of dbt_utils, but as for now, we cannot do it because of dependency on fivetran_utils. Is there a timeline to upgrade the version of dbt_utils used? I imagine this is also causing problems for existing Fivetran packages users too, so it shouldn't be a problem only for us :)

[fivetran-joemarkiewicz]

Hi @mateuszklimek thanks for raising this! This is something we noticed the other month and quickly found that making the breaking change updates across 80+ repos was going to be a large effort 🤯. Therefore, we are currently in the process of updating our fivetran_utils package (and other packages) and plan to have full support at the end of this month! You can expect fivetran_utils v0.4.0 to have the following dependency on dbt-utils:

packages:
  - package: dbt-labs/dbt_utils
    version: [">=0.9.0", "<1.1.0"]

You can see the open PR #85 with our soon to be released updates.

That being said, if you are only relying on fivetran_utils, you should see a release for this package coming in the next week or so. Therefore, you can hopefully reference v0.4.0 of fivetran_utils sooner! With the caveat being non of our other packages will reference this version of fivetran_utils until the end of the month.

[pkanter]

@fivetran-joemarkiewicz , thanks for the update.

[pkanter]

@fivetran-joemarkiewicz , are you still on schedule for having this updated by end-of-month?

[fivetran-joemarkiewicz]

Hi @pkanter, the latest release (v0.4.0) of Fivetran Utils is now live! We are currently in the process of rolling out the dependency updates to all of our packages as we speak. Our hope is to have all of our packages update by the end of the month.

However, we will have to wait until dbt-utils v1.0.0 is live before releasing any updates. Last I heard, this should be live around the same timeline. I will plan to share more once dbt-utils latest release is live and our packages have all been updated. Best case, we are able to have all packages done by the end of the month. Worse case it is mid November.

[tinomen]

It would have been great to see a v0.3.10 come out that supported dbt_utils [">=0.9.0", "<1.0.0"] so we could use this with many other packages

[fivetran-joemarkiewicz]

Hi @tinomen thanks for reaching out. Our initial plan was to release a patch release to support a pre v1.0.0 dbt-utils version. However, due to the require-dbt-version within the v0.9.0 range of dbt-utils we were not able to release a patch update.

Our v0.3.x version of fivetran-utils (and the dependent packages) have a supported dbt-core range of [">=1.0.0", "<2.0.0"]. Therefore, if we were to cut a release to support the v0.9.0 release then we could inadvertently break a large number of our users dbt projects. 😞 As a result, we would need to cut a breaking change of fivetran-utils and all our upstream dependent packages (over 80 of them). Since it seemed dbt-utils v1.0.0 was right on the horizon, we opted to skip 0.9.0 and prepare for v1.0.0. Once v1.0.0 is released, we will be able to roll out updates across all our packages.

I apologize for the inconvenience this dependency restriction has caused. We are actually actively exploring the possibility of removing a dependency on dbt-utils across our packages so this dependency restriction won't happen in the future.

[tinomen]

I know that the horrible package dep issue in dbt is not your problem, which is why it would be great to see every significant package put out a version that supports each minor version of the package they depend on. dbt_utils in particular is annoying.

I'm looking at the recent v0.3.9 release and it has the package defined as

packages:
  - package: dbt-labs/dbt_utils
    version: [">=0.8.0", "<0.9.0"]

So you can see why it would be great to see a v0.3.10.

I don't see any 3.x releases with a [">=1.0.0", "<2.0.0"] package dep. Can you point me to one.

[fivetran-joemarkiewicz]

Unfortunately, all of our source packages (dbt_zendesk_source being an example) have a range dependency on this fivetran_utils repo.

Therefore, if we were to cut a v0.3.10 release of our fivetran_utils package that includes the [">=0.9.0", "<1.0.0"] or [">=1.0.0", "<2.0.0"] range, then it would bring with it a number of breaking changes to all of our package users without warning. I mention this because this did happen back when dbt-core v0.20.0 and dbt-utils v0.7.0 were released 😞. Due to this, we need to cut a breaking release (v0.4.0 in this case) whenever we bump the dbt-utils (or other package deps) by either a major (X.0.0) or minor (0.X.0) release.

As a result, I agree with you that it would be great to include the patch release you mentioned, but the risk of introducing a breaking change across all our packages is prohibiting us from moving forward.

What I can point to however, is the the v0.4.0 release that I mentioned which will offer support for dbt v1.0.0. If you wanted to use this version and one of our packages in the meantime, you can leverage the MagicBot/dbt-utils-cross-db-migration branch (see our dbt_quickbooks_source open PR as an example) on any of our package repos. This branch contains all the working migration changes for dbt-utils v1.0.0 (and the changes brought within v0.9.0). You could use this version of the package in the meantime until dbt-utils v1.0.0 is released.

I again apologize for this dependency dance and completely understand how frustrating it is. I really appreciate you raising this with us as we have heard from others about this frustration. I am going to investigate over the next week or so to see what it would look like for us to completely remove the dbt-utils dependency for our next round of breaking changes across our packages.

[tinomen]

I can't move up to v1 until the other packages we use upgrade. A couple are planning to remove the dbt_utils dependency so that will be a great step forward as well.

Thank you for the github package.

[fivetran-joemarkiewicz]

If you would be willing to share which packages you are using, I can try and get a branch out that removes the dependency for you to test and use in the interim.

[tinomen]

dbt_date is the big one that we'd like to upgrade, but they have a plan to remove dbt_utils

[fivetran-joemarkiewicz]

My apologies, which Fivetran dbt packages are you using as we may be able to use those as the test in this case.

[tinomen]

Haha, sorry about the misunderstanding. We are using github, jira_source, and salesforce_source

[fivetran-joemarkiewicz]

Hi all, sorry for the late reply to this issue, but just wanted to share that in December we were able to cut a release of Fivetran Utils that uses the latest versions of dbt-utils. As such, closing this issue.

[pkanter]

Thanks for the update, Joe. *From:* Joe Markiewicz ***@***.***> *Sent:* Wednesday, June 7, 2023 11:26 AM *To:* fivetran/dbt_fivetran_utils ***@***.***> *Cc:* Paul Kanterman ***@***.***>; Mention < ***@***.***> *Subject:* Re: [fivetran/dbt_fivetran_utils] upgrade dbt_utils dependency (Issue #90) Closed #90 <#90> as completed. — Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFKLZQ3O2WDNADJ3SNWM4PTXKCMRXANCNFSM6AAAAAAQ3LSLWA> . You are receiving this because you were mentioned.Message ID: < ***@***.***>