Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] --empty flag should be exposed in flags variable #10152

Closed
2 tasks done
jlucas91 opened this issue May 15, 2024 · 7 comments · Fixed by #10315
Closed
2 tasks done

[Feature] --empty flag should be exposed in flags variable #10152

jlucas91 opened this issue May 15, 2024 · 7 comments · Fixed by #10315
Labels
enhancement New feature or request

Comments

@jlucas91
Copy link

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When running dbt 1.8 EMPTY is not present on the flags variable.

Expected Behavior

When using the --empty flag, I would expect EMPTY to be set to true. This is would bring it in line with other first class flags like full-refresh and fail-fast.

Steps To Reproduce

  1. ./dbt run --empty
  2. Within a model, include {{ print(flags) }}
  3. FLAGS is not present

Relevant log output

No response

Environment

- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?

snowflake

Additional Context

It does not appear to be included in the flags module https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/flags.py

@jlucas91 jlucas91 added bug Something isn't working triage labels May 15, 2024
@dbeatty10 dbeatty10 self-assigned this May 16, 2024
@dbeatty10
Copy link
Contributor

Thanks for reaching out @jlucas91 !

This sounds more like a feature request to me. Can you share more why this seems like a bug to you?

@dbeatty10 dbeatty10 removed their assignment May 16, 2024
@jlucas91
Copy link
Author

I think it could go either way and no worries on my end if you'd prefer to reclassify.

It felt like a potential unintentional miss in the --empty flag release to me, which is why I leaned towards calling it a bug. While the documentation does state the code is the source of truth for what flags are included in the variable, it seemed strange for a significant and first class flag to be omitted from the interface.

@jtcohen6
Copy link
Contributor

jtcohen6 commented May 16, 2024

For what it's worth, I do buy this as a thing we should do!

Among other things, this might enable us to provide users with slightly more flexibility & customization while working around known edge cases of --empty.

As an example:

@jtcohen6 jtcohen6 added enhancement New feature or request and removed bug Something isn't working triage labels May 16, 2024
@jtcohen6 jtcohen6 changed the title [Bug] --empty flag is not exposed in flags variable [Bug] --empty flag should be exposed in flags variable May 16, 2024
@dbeatty10 dbeatty10 changed the title [Bug] --empty flag should be exposed in flags variable [Feature] --empty flag should be exposed in flags variable May 16, 2024
@jlucas91
Copy link
Author

jlucas91 commented May 17, 2024

That's exactly how I bumped into the flags limitation @jtcohen6 ! I'm doing a lot of checks like this right now to guard the various limitation cases we've bumped into:

{% set empty_run = '--empty' in flags.INVOCATION_COMMAND %}
{% if empty_run %}
-- compatibility code
{% else %}
-- model code
{% endif %}

That's pretty gross and will get nicer with the flags change. To take it one step further we've opted to implement a is_empty_run macro to clean up the syntax even more. A similar macro might be nice to roll into core DBT as a bit of syntactic sugar for users (similar to is_incremntal()).

@dsillman2000
Copy link

That's exactly how I bumped into the flags limitation @jtcohen6 ! I'm doing a lot of checks like this right now to guard the various limitation cases we've bumped into:

{% set empty_run = '--empty' in flags.INVOCATION_COMMAND %}
{% if empty_run %}
-- compatibility code
{% else %}
-- model code
{% endif %}

That's pretty gross and will get nicer with the flags change. To take it one step further we've opted to implement a is_empty_run macro to clean up the syntax even more. A similar macro might be nice to roll into core DBT as a bit of syntactic sugar for users (similar to is_incremntal()).

Hi @jlucas91 - this feature has a crucial bug (flags.INVOCATION_COMMAND) which prevents using it reliably the way you have above.

If you run dbt using programmatic invocation, then the flags.INVOCATION_COMMAND routes sys.argv directly. I'm seeing this when I invoke dbt programmatically from a pytest function:

def test_not_empty():
    dbt_runner = dbtRunner()
    run_result = dbt_runner.invoke(
        ["compile", "--select", "compile_empty_test"],
        project_dir=DBT_PROJECT_DIR,
        profiles_dir=DBT_PROJECT_DIR,
    )
    results = run_result.result.results
    assert len(results) == 1
    assert results[0].node.compiled_code.strip().lower() == "select 2"

In the model I "info-print" the flags.INVOCATION_COMMAND which reveals it harvested sys.argv from the pytest invocation:

INVOCATION_COMMAND='dbt -sv test_empty.py'

This unrelated bug prevents us from reliably using flags.INVOCATION_COMMAND for this purpose (or virtually anything else, so long as we're programmatically invoking dbt, e.g. in CI). Any other work-arounds available until "EMPTY" is added to flags?

(P.S. - I didn't manage to find this bug mentioned anywhere - should I open a separate issue to have it addressed? Thanks!)

@dbeatty10
Copy link
Contributor

(P.S. - I didn't manage to find this bug mentioned anywhere - should I open a separate issue to have it addressed? Thanks!)

@dsillman2000 Yes, that would be great if you open a separate issue 👍

@dbeatty10
Copy link
Contributor

Resolved by #10315

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants