Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vdk-core: KeyError in user code coming from pandas classified as PlatformError #2839

Closed
DeltaMichael opened this issue Oct 26, 2023 · 0 comments · Fixed by #2840
Closed

vdk-core: KeyError in user code coming from pandas classified as PlatformError #2839

DeltaMichael opened this issue Oct 26, 2023 · 0 comments · Fixed by #2840
Assignees
Labels
bug Something isn't working initiative: VDK Run Logs

Comments

@DeltaMichael
Copy link
Contributor

DeltaMichael commented Oct 26, 2023

Steps To Reproduce

Run a job in vdk with the following step

import logging
import pandas as pd

from vdk.api.job_input import IJobInput

log = logging.getLogger(__name__)

def run(job_input: IJobInput):
    """
    Function named `run` is required in order for a python script to be recognized as a Data Job Python step and executed.

    VDK provides to every python step an object - job_input - that has methods for:

    * executing queries to OLAP Database;
    * ingesting data into a database;
    * processing data into a database.
    See IJobInput documentation for more details.
    """
    log.info(f"Starting job step {__name__}")
    df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],
                    'value': [1, 2, 3, 5]})
    df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],
                    'value': [5, 6, 7, 8]})
    # df1.merge(df2, left_on='lkey', right_on='rkey')
    result = pd.merge(df1, df2, how="inner", on=["timestamp", "status_code", "response_status", "exception", "region", "region_type", "tenant", "utc_time"])

Expected behavior
KeyError is thrown and job ends with UserError.

Actual behavior
KeyError is thrown and job ends with PlatformError

@DeltaMichael DeltaMichael added the bug Something isn't working label Oct 26, 2023
@DeltaMichael DeltaMichael self-assigned this Oct 26, 2023
@DeltaMichael DeltaMichael linked a pull request Oct 26, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working initiative: VDK Run Logs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant