Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed Readme typo #29

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

Hiyorimi
Copy link

@Hiyorimi Hiyorimi commented Nov 21, 2017

Well, actually I just wanted to report an issue and I'm not sure that I got how the demo notebooks load.

On the official tutorial page there is a link to get all the Spark notebooks with hash 2CJW53M52. I suppose that this page was created from GitHub md file.

However, on both of this pages screenshot suggest that after download of test data, data is moved from local system to '/tmp' folder in the Hadoop like this:

hdfs dfs -rm -r -f /tmp/flights.csv

# put data into HDFS
hdfs dfs -put /tmp/flights.csv /tmp/

and later:


// Create a flights DataFrame from CSV file
val flights = spark.read
              .option("header", "true")                              // Use first line as header
              .option("inferSchema", "true")                         // Infer schema
              .csv("/tmp/airflightsdelays/flights.csv")                               // Read data

Which is not correct. The error is present only at screenshots, the code in the notebook seems to be ok.

In the original repo there is a folder with what seems to be a correct notebook (with hash 2C174C9EK) for this tutorial, with the commands:

hadoop fs -mkdir /tmp/airflightsdelays
# put data into HDFS
hadoop fs -put /tmp/flights.csv /tmp/airflightsdelays/

So it might be a good idea to fix either a screenshot or the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant