[CT-3144] [Regression] build never starts on 1.6.2 and 1.6.3 #8692
Comments
github-actions
bot
changed the title
dbeatty10
changed the title
|
HI! I am curious about this line of your output: If you try doing a |
Hi, Gracie! Thanks for checking! I just ran the with the flag as requested on Meanwhile, I'm testing the same with |
|
@renanleme Would you be willing to do the following?
It might give us a hint as to what is going on so that we can reproduce the issue. |
|
Hi, @peterallenwebb Sure, it was running for more than 1 hour, here is the stack trace My guess is probably some issue with the recursion when excluding some models as we are using exclusion on our selector |
|
I strongly suspect a performance regression associated with this changeset: f1dddaa @renanleme It's really asking you to go above and beyond, but if you can revert the changes in this changeset on dbt/graph/graph.py and re-run in 1.6.3, that would prove our case. |
Sure, I will try that in a bit |
Hey, @peterallenwebb Looks like that's the issue. I reverted this one: f1dddaa And it worked 🎉 Let me know if I can help debugging more. |
|
@renanleme Thanks for your extremely fast responses! I'll let the team know and we'll get a fix released as soon as we can. |
You are welcome! I was testing and I think this this code below solves the issue here: The amount of models are the same so it should be ok (I ran some tests, all passed, including Thanks @peterallenwebb ! |
graciegoheen
added
the
Highest Severity
|
Update on my progress. I created a quick random graph generator to test the relative performance of the Graph.ancestors function under some hopefully-reasonable assumptions. This is the script: from dbt.graph.graph import Graph
import time
import networkx
import random
def rand_edge(u, v, prob_test_parent):
return (u,v,{'edge_type': 'parent_test'}) if random.random() < prob_test_parent else (u, v)
def rand_dag(num_nodes, exp_out_degree, prob_test_parent, seed):
complete_edge_count = num_nodes * (num_nodes-1) / 2.0
edge_prob = exp_out_degree * num_nodes / complete_edge_count
g = networkx.gnp_random_graph(num_nodes, edge_prob, seed, directed=True)
dag = networkx.DiGraph([rand_edge(u, v, prob_test_parent) for (u,v) in g.edges() if u < v])
return dag
def time_ancestors(g, test_nodes):
start_time = time.perf_counter()
for node in test_nodes:
a = g.ancestors(node)
stop_time = time.perf_counter()
return stop_time - start_time
seed = 12345
g100 = Graph(rand_dag(100, 10.0, 0.15, seed))
g400 = Graph(rand_dag(400, 10.0, 0.15, seed))
g1600 = Graph(rand_dag(1600, 10.0, 0.15, seed))
g3600 = Graph(rand_dag(3600, 10.0, 0.15, seed))
print("g100: " + str(time_ancestors(g100, [10, 20, 40, 60, 80])))
print("g400: " + str(time_ancestors(g400, [40, 80, 160, 240, 320])))
print("g1600: " + str(time_ancestors(g1600, [160, 320, 640, 960, 1290])))
print("g3600: " + str(time_ancestors(g3600, [300, 600, 1000, 2000, 3200])))The results on some different branches: ObservationsAssuming the random graphs are a reasonable approximation of the real world, we regressed the performance of this function in 1.6.0, and it now takes forty times longer to execute. However, even with the currently proposed fix PR, the performance is still much worse (ten times longer) than in 1.5.latest. It's possible that on real-world data the PR is a better fix than these tests indicate, but it's hard to know, so I'm working on a more fundamental fix. |
|
reopening until backport is complete |
|
@renanleme Your contribution has been merged and backported. It will appear soon in dbt 1.6.4. Thank you for contributing to dbt-core! |
Is this a regression in a recent version of dbt-core?
Current Behavior
Afterm upgrading to
1.6.2from1.5.1our build is taking way longer to start the first modelFrom around 7 to 8 minutes to complete to more than 1 hour without starting.
I'm running
dbt build --selector XModels:
Selector example
Expected/Previous Behavior
That the build would take around 8 minutes to start after compiling all the models
Steps To Reproduce
Takes around 6 to 7 minutes
Same command, different version
And it goes for more than 50 minutes without starting with CPU at 100%
Relevant log output
Checking the log file there's nothing being logged after `Found 2840 models...` line besides the CPU usage at 100%Environment
Which database adapter are you using with dbt?
snowflake
Additional Context
I thought that this could be related to this issue: [CT-2960] [Bug] Compile performance hit (DBT 1.4.6 -> 1.6.0) #8360
But I upgraded to 1.6.3 before posting here and the issue continues
I ran
dbt compile --selector XYZon both and both ran successfullyThe text was updated successfully, but these errors were encountered: