Skip to content

Commit

Permalink
SPARK-6992 : Fix documentation example for Spark SQL on StructType
Browse files Browse the repository at this point in the history
This patch is fixing the Java examples for Spark SQL when defining
programmatically a Schema and mapping Rows.

Author: Olivier Girardot <o.girardot@lateral-thoughts.com>

Closes apache#5569 from ogirardot/branch-1.3 and squashes the following commits:

c29e58d [Olivier Girardot] SPARK-6992 : Fix documentation example for Spark SQL on StructType

(cherry picked from commit c9b1ba4)
Signed-off-by: Reynold Xin <rxin@databricks.com>
  • Loading branch information
Olivier Girardot authored and rxin committed Apr 18, 2015
1 parent d850b4b commit 5f095d5
Showing 1 changed file with 8 additions and 5 deletions.
13 changes: 8 additions & 5 deletions docs/sql-programming-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -555,13 +555,16 @@ by `SQLContext`.

For example:
{% highlight java %}
// Import factory methods provided by DataType.
import org.apache.spark.sql.types.DataType;
import org.apache.spark.api.java.function.Function;
// Import factory methods provided by DataTypes.
import org.apache.spark.sql.types.DataTypes;
// Import StructType and StructField
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.types.StructField;
// Import Row.
import org.apache.spark.sql.Row;
// Import RowFactory.
import org.apache.spark.sql.RowFactory;

// sc is an existing JavaSparkContext.
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
Expand All @@ -575,16 +578,16 @@ String schemaString = "name age";
// Generate the schema based on the string of schema
List<StructField> fields = new ArrayList<StructField>();
for (String fieldName: schemaString.split(" ")) {
fields.add(DataType.createStructField(fieldName, DataType.StringType, true));
fields.add(DataTypes.createStructField(fieldName, DataTypes.StringType, true));
}
StructType schema = DataType.createStructType(fields);
StructType schema = DataTypes.createStructType(fields);

// Convert records of the RDD (people) to Rows.
JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
return Row.create(fields[0], fields[1].trim());
return RowFactory.create(fields[0], fields[1].trim());
}
});

Expand Down

0 comments on commit 5f095d5

Please sign in to comment.