From a0b17ead729511a856609a6ac80033d30f1dce71 Mon Sep 17 00:00:00 2001 From: CHEN LIANG Date: Sat, 15 Aug 2015 00:04:08 -0700 Subject: [PATCH 1/3] Some updates for programming_guide.md Some updates for the section "Program Skeleton" of programming_guide.md --- docs/apis/programming_guide.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/apis/programming_guide.md b/docs/apis/programming_guide.md index 85c639ec88940..22600c862e864 100644 --- a/docs/apis/programming_guide.md +++ b/docs/apis/programming_guide.md @@ -217,13 +217,11 @@ programs with a `main()` method. Each program consists of the same basic parts: 1. Obtain an `ExecutionEnvironment`, 2. Load/create the initial data, 3. Specify transformations on this data, -4. Specify where to put the results of your computations, and +4. Specify where to put the results of your computations, 5. Trigger the program execution We will now give an overview of each of those steps, please refer to the respective sections for -more details. Note that all -{% gh_link /flink-java/src/main/java/org/apache/flink/api/java "core classes of the Java API" %} -are found in the package `org.apache.flink.api.java`. +more details. Note that all core classes of the Java API are found in the package {% gh_link /flink-java/src/main/java/org/apache/flink/api/java "org.apache.flink.api.java" %}. The `ExecutionEnvironment` is the basis for all Flink programs. You can obtain one using these static methods on class `ExecutionEnvironment`: @@ -231,6 +229,8 @@ obtain one using these static methods on class `ExecutionEnvironment`: {% highlight java %} getExecutionEnvironment() +createCollectionsEnvironment() + createLocalEnvironment() createLocalEnvironment(int parallelism) createLocalEnvironment(Configuration customConfiguration) @@ -272,7 +272,7 @@ a map transformation looks like this: {% highlight java %} DataSet input = ...; -DataSet tokenized = text.map(new MapFunction() { +DataSet tokenized = input.map(new MapFunction() { @Override public Integer map(String value) { return Integer.parseInt(value); @@ -284,14 +284,16 @@ This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can either write the result -to a file system (HDFS or local) or print it. +Once you have a DataSet containing your final results, you can write the result +to a file system (HDFS or local) or socket or print it. {% highlight java %} writeAsText(String path) writeAsCsv(String path) write(FileOutputFormat outputFormat, String filePath) +writeToSocket(String hostName, int port, SerializationSchema schema) + print() printOnTaskManager() @@ -307,10 +309,10 @@ programs with a `main()` method. Each program consists of the same basic parts: 1. Obtain an `ExecutionEnvironment`, 2. Load/create the initial data, 3. Specify transformations on this data, -4. Specify where to put the results of your computations, and +4. Specify where to put the results of your computations, 5. Trigger the program execution -We will now give an overview of each of those steps but please refer to the respective sections for +We will now give an overview of each of those steps, please refer to the respective sections for more details. Note that all core classes of the Scala API are found in the package {% gh_link /flink-scala/src/main/scala/org/apache/flink/api/scala "org.apache.flink.api.scala" %}. @@ -324,6 +326,8 @@ def getExecutionEnvironment def createLocalEnvironment(parallelism: Int = Runtime.getRuntime.availableProcessors())) def createLocalEnvironment(customConfiguration: Configuration) +def createCollectionsEnvironment + def createRemoteEnvironment(host: String, port: String, jarFiles: String*) def createRemoteEnvironment(host: String, port: String, parallelism: Int, jarFiles: String*) {% endhighlight %} @@ -361,15 +365,15 @@ a map transformation looks like this: {% highlight scala %} val input: DataSet[String] = ... -val mapped = text.map { x => x.toInt } +val mapped = input.map { x => x.toInt } {% endhighlight %} This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can either write the result -to a file system (HDFS or local) or print it. +Once you have a DataSet containing your final results, you can write the result +to a file system (HDFS or local) or socket or print it. {% highlight scala %} def writeAsText(path: String, writeMode: WriteMode = WriteMode.NO_OVERWRITE) @@ -382,6 +386,8 @@ def write(outputFormat: FileOutputFormat[T], path: String, writeMode: WriteMode = WriteMode.NO_OVERWRITE) +def writeToSocket(hostname: String, port: Integer, schema: SerializationSchema[T, Array[Byte]]) + def printOnTaskManager() def print() From b0bd938146acf9d013d61d6c3b3c4bdc1ccd7138 Mon Sep 17 00:00:00 2001 From: CHEN LIANG Date: Sat, 15 Aug 2015 10:38:47 -0700 Subject: [PATCH 2/3] Update programming_guide.md --- docs/apis/programming_guide.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/apis/programming_guide.md b/docs/apis/programming_guide.md index 22600c862e864..039590babe9bf 100644 --- a/docs/apis/programming_guide.md +++ b/docs/apis/programming_guide.md @@ -284,16 +284,13 @@ This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can write the result -to a file system (HDFS or local) or socket or print it. +Once you have a DataSet containing your final results, you can either write the result to a file system (HDFS or local) or print it. {% highlight java %} writeAsText(String path) writeAsCsv(String path) write(FileOutputFormat outputFormat, String filePath) -writeToSocket(String hostName, int port, SerializationSchema schema) - print() printOnTaskManager() @@ -372,8 +369,7 @@ This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can write the result -to a file system (HDFS or local) or socket or print it. +Once you have a DataSet containing your final results, you can either write the result to a file system (HDFS or local) or print it. {% highlight scala %} def writeAsText(path: String, writeMode: WriteMode = WriteMode.NO_OVERWRITE) @@ -386,8 +382,6 @@ def write(outputFormat: FileOutputFormat[T], path: String, writeMode: WriteMode = WriteMode.NO_OVERWRITE) -def writeToSocket(hostname: String, port: Integer, schema: SerializationSchema[T, Array[Byte]]) - def printOnTaskManager() def print() From 38467ef0694e1badf25b2045bd75855cea2a5f80 Mon Sep 17 00:00:00 2001 From: CHEN LIANG Date: Sat, 15 Aug 2015 14:09:34 -0700 Subject: [PATCH 3/3] Update programming_guide.md --- docs/apis/programming_guide.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/apis/programming_guide.md b/docs/apis/programming_guide.md index 039590babe9bf..eb874b9f3e1a8 100644 --- a/docs/apis/programming_guide.md +++ b/docs/apis/programming_guide.md @@ -284,7 +284,8 @@ This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can either write the result to a file system (HDFS or local) or print it. +Once you have a DataSet containing your final results, you can either write the result +to a file system (HDFS or local) or print it. {% highlight java %} writeAsText(String path) @@ -369,7 +370,8 @@ This will create a new DataSet by converting every String in the original set to an Integer. For more information and a list of all the transformations, please refer to [Transformations](#transformations). -Once you have a DataSet containing your final results, you can either write the result to a file system (HDFS or local) or print it. +Once you have a DataSet containing your final results, you can either write the result +to a file system (HDFS or local) or print it. {% highlight scala %} def writeAsText(path: String, writeMode: WriteMode = WriteMode.NO_OVERWRITE)