Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider creating spark_read_csv_options() #2408

Open
javierluraschi opened this issue Apr 11, 2020 · 0 comments
Open

Consider creating spark_read_csv_options() #2408

javierluraschi opened this issue Apr 11, 2020 · 0 comments
Assignees

Comments

@javierluraschi
Copy link
Collaborator

@javierluraschi javierluraschi commented Apr 11, 2020

Maybe worth considering adding a spark_options_csv(sep = NULL, encoding = NULL, etc) helper function to properly document all the options users can use under spark_read_csv(), see https://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/sql/DataFrameReader.html

So one could write,

spark_read_csv(sc, path = "file.csv", options = spark_options_csv(maxColumns = 100000L))

See also https://community.rstudio.com/t/spark-r-setting-max-columns/60908.

@yitao-li yitao-li added this to the 1.4.0 milestone Jul 30, 2020
@yitao-li yitao-li self-assigned this Jul 30, 2020
@yitao-li yitao-li modified the milestones: 1.4.0, 1.5.0 Sep 4, 2020
@yitao-li yitao-li modified the milestones: 1.5.0, 1.6.0 Dec 3, 2020
@yitao-li yitao-li removed this from the 1.6.0 milestone Feb 23, 2021
@yitao-li yitao-li assigned edgararuiz-zz and unassigned yitao-li Jan 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants