Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-16542][SQL][PYSPARK] Fix bugs about types that result an array of null when creating dataframe using python #14198

Closed
wants to merge 10 commits into from

Commits on Jul 9, 2016

  1. use array.typecode to infer type

    Python's array has more type than python it self, for example
    python only has float while array support 'f' (float) and 'd' (double)
    Switching to array.typecode helps spark make a better inference
    
    For example, for the code:
    
    from pyspark.sql.types import _infer_type
    from array import array
    a = array('f',[1,2,3,4,5,6])
    _infer_type(a)
    
    We will get ArrayType(DoubleType,true) before change,
    but ArrayType(FloatType,true) after change
    zasdfgbnm committed Jul 9, 2016
    Configuration menu
    Copy the full SHA
    a127486 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    70131f3 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2016

  1. sync with upstream

    zasdfgbnm committed Jul 13, 2016
    Configuration menu
    Copy the full SHA
    505e819 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2016

  1. Configuration menu
    Copy the full SHA
    05979ca View commit details
    Browse the repository at this point in the history
  2. sync with upstream

    zasdfgbnm committed Jul 14, 2016
    Configuration menu
    Copy the full SHA
    5cd817a View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2016

  1. Configuration menu
    Copy the full SHA
    cd2ec6b View commit details
    Browse the repository at this point in the history
  2. merge with upstream/master

    zasdfgbnm committed Oct 17, 2016
    Configuration menu
    Copy the full SHA
    527d969 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    82223c0 View commit details
    Browse the repository at this point in the history
  4. fix code style

    zasdfgbnm committed Oct 17, 2016
    Configuration menu
    Copy the full SHA
    0a967e2 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2016

  1. Configuration menu
    Copy the full SHA
    2059435 View commit details
    Browse the repository at this point in the history