perf(workflows): Avoid ~0.6s regression in backported OrganizationCombinedRuleIndexEndpoint.get

[kcons]

The JIT is expensive and being used when we don't need it. We can try to work around it or restructure the query to avoid the jit threshold, but just scoped disabling gets us exactly what we want.

I've confirmed the JIT impact by analyzing sample queries and observing that the vast majority of the time is spent in JIT, but with JIT disabled the query is much faster.
Demo at https://redash.getsentry.net/queries/10853

I've also confirmed in traces that this query is consistently slow (280ms-450ms) and issued twice per request.

NB: The workflow engine variant of this endpoint isn't yet released, so the risk here should be quite low.

[github-actions]

Backend Test Failures

Failures on 4e05f70 in this run:

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexEndpointTest::test_legacy_models_header — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:104: in test_legacy_models_header
    resp = self.get_success_response(self.organization.slug)
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexWorkflowEngineTest::test_workflow_engine_dataset_filtering — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1511: in test_workflow_engine_dataset_filtering
    response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexWorkflowEngineTest::test_workflow_engine_dual_written_rules — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1427: in test_workflow_engine_dual_written_rules
    response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexWorkflowEngineTest::test_workflow_engine_endpoint_returns_successfully — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1412: in test_workflow_engine_endpoint_returns_successfully
    response = self.get_success_response(self.organization.slug, project=[self.project.id])
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexWorkflowEngineTest::test_workflow_engine_filtering_by_name — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1482: in test_workflow_engine_filtering_by_name
    response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexWorkflowEngineTest::test_workflow_engine_filtering_by_team — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1465: in test_workflow_engine_filtering_by_team
    response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexParityTest::test_dual_written_rules_parity — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1563: in test_dual_written_rules_parity
    we_response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py::OrganizationCombinedRuleIndexParityTest::test_filtering_parity — log

tests/sentry/incidents/endpoints/test_organization_combined_rule_index_endpoint.py:1611: in test_filtering_parity
    we_response = self.get_success_response(
src/sentry/testutils/cases.py:631: in get_success_response
    assert_status_code(response, status.HTTP_200_OK)
src/sentry/testutils/asserts.py:47: in assert_status_code
    assert minimum <= response.status_code < maximum, (
E   AssertionError: (500, b'{"detail":"Internal Error","errorId":null}')
E   assert 500 < 201
E    +  where 500 = <Response status_code=500, "application/json">.status_code

[ceorourke]

Approving based on your explanation and confidence in your analysis with the caveat that I am not that well versed in the Django internals. Feel free to wait for Dan's final approval.

[wedamija]

Why can't we use a transaction here? It won't apply to the crons db of course, but we can start a transaction in the default db and then just disable jit using local. That reduces the risk of us accidentally leaking this by setting it at the connection level

[kcons]

As you can see in the test failures above, it runs afoul of enforce_no_cross_transaction_interactions (

sentry/tests/conftest.py

Lines 94 to 99 in 28331f1

	@pytest.fixture(autouse=True)
	def setup_enforce_monotonic_transactions(request: pytest.FixtureRequest) -> Generator[None]:
	from sentry.testutils.hybrid_cloud import enforce_no_cross_transaction_interactions
	with enforce_no_cross_transaction_interactions():
	yield

)

[wedamija]

That's fine, you can disable that. We unfortunately have to do this a lot when working with monitors, with in_test_hide_transaction_boundary():.