Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix materializing upstream outputs for dbt #3617

Merged
merged 2 commits into from
Sep 29, 2023
Merged

Fix materializing upstream outputs for dbt #3617

merged 2 commits into from
Sep 29, 2023

Conversation

christopherscholz
Copy link
Contributor

@christopherscholz christopherscholz commented Sep 28, 2023 •
edited

Description

The output of a block was not normalized to an pandas.DataFrame and wasn't checked whether this df is empty. Which caused errors in upstream blocks, which output wasn't already an df.

I have normalized the output and made sure only non empty df are materialized.

How Has This Been Tested?

  • Checked manually

Checklist

  • The PR is tagged with proper labels (bug, enhancement, feature, documentation)
  • I have performed a self-review of my own code
  • I have added unit tests that prove my fix is effective or that my feature works
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation

cc: @wangxiaoyou1993 @tommydangerous

@christopherscholz christopherscholz marked this pull request as ready for review September 28, 2023 10:32
@wangxiaoyou1993 wangxiaoyou1993 added the bug Something isn't working label Sep 28, 2023
wangxiaoyou1993
wangxiaoyou1993 reviewed Sep 29, 2023
View reviewed changes
mage_ai/data_preparation/models/block/__init__.py Outdated
df = pd.DataFrame()

if df.empty:
logger.info('No data for dbt to materialize.')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logger can be None

@wangxiaoyou1993 wangxiaoyou1993 merged commit 911d316 into mage-ai:master Sep 29, 2023
5 checks passed
@wangxiaoyou1993 wangxiaoyou1993 mentioned this pull request Oct 3, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangxiaoyou1993 wangxiaoyou1993 wangxiaoyou1993 approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@christopherscholz @wangxiaoyou1993