Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: When collected, long values are cast to int #1155

Open
aesteve opened this issue Jul 27, 2023 · 0 comments
Open

[BUG]: When collected, long values are cast to int #1155

aesteve opened this issue Jul 27, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@aesteve
Copy link

aesteve commented Jul 27, 2023

Describe the bug
When running DataFrame.Collect() a long value in the dataset (even with a proper schema set to LongType) can be collected as an int.

To Reproduce

This reproduces the issue:

        long value = 1;
        string fieldName = "SomeLongValue"; 
        var df = _spark.CreateDataFrame(
            new[]
            {
                new GenericRow(new object[] { value } ),
            },
            new StructType(new [] { new StructField(fieldName, new LongType(), false)})
        );
        var collected = df.Collect().First().Get(fieldName);
        Assert.Equal(value, collected); // this fails since `value` is a long whereas `collected` is an int

On the other hand, this test would run fine:

        long value = long.MaxValue;
        string fieldName = "SomeLongValue"; 
        var df = _spark.CreateDataFrame(
            new[]
            {
                new GenericRow(new object[] { value } ),
            },
            new StructType(new [] { new StructField(fieldName, new LongType(), false)})
        );
        var collected = df.Collect().First().Get(fieldName);
        Assert.Equal(value, collected); // this works

It might come from Unpickling or from Spark -> Spark-Python, I have no idea.

Expected behavior

When the Schema specifies a column is of LongType, the collected value should be a long

Desktop (please complete the following information):

  • OS: Windows 11
  • Version: 2.1.1
@aesteve aesteve added the bug Something isn't working label Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant