CT-3144 Fix test edges type filter on Graph

[renanleme]

resolves #8692

Problem

After upgrading to 1.6.2 from 1.5.1 our build is taking way longer to start the first model
From around 7 to 8 minutes to complete to more than 1 hour without starting.

Solution

Changed the networkx function used to filter test type edges

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
• This PR includes type annotations for new and modified functions

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (aa86fdf) 86.61% compared to head (7c36572) 65.16%.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #8696       +/-   ##
===========================================
- Coverage   86.61%   65.16%   -21.46%     
===========================================
  Files         176      176               
  Lines       25678    25681        +3     
===========================================
- Hits        22240    16734     -5506     
- Misses       3438     8947     +5509     

Flag	Coverage Δ
integration	?
unit	65.16% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
core/dbt/graph/graph.py	86.48% <100.00%> (-10.70%)	⬇️

... and 117 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[peterallenwebb]

@renanleme Thanks again for your work here.

I found that there were a couple remaining problems, which can be resolved by using this definition of filter_edges_by_type in your PR:

    def filter_edges_by_type(self, first_node, second_node, edge_type):
        return self.graph.get_edge_data(first_node, second_node).get("edge_type") != edge_type

I found that the use of get_edge_data() is considerably faster than the double index operation it replaces, and the != comparison fixes a simple typo.

If you are able to make this change in your PR and sign the CLA, I will merge it and get it backported so that all dbt users can benefit!

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @renanleme

[renanleme]

@renanleme Thanks again for your work here.

I found that there were a couple remaining problems, which can be resolved by using this definition of filter_edges_by_type in your PR:

    def filter_edges_by_type(self, first_node, second_node, edge_type):
        return self.graph.get_edge_data(first_node, second_node).get("edge_type") != edge_type

I found that the use of get_edge_data() is considerably faster than the double index operation it replaces, and the != comparison fixes a simple typo.

If you are able to make this change in your PR and sign the CLA, I will merge it and get it backported so that all dbt users can benefit!

Hey, @peterallenwebb, thanks for helping me with it!

I updated the pr as suggested, I tested my use case locally and it's a little bit faster indeed.

Let me know if anything else is needed.