APEXMALHAR-2463 FTP input operator sample app, and documentation #596

francisf · 2017-03-30T04:42:48Z

amberarrow · 2017-04-25T03:31:42Z

Ok, will do -- give me a couple of days.

chaithu14

Minor comment in application, rest looks good to me.

chaithu14 · 2017-04-26T10:20:27Z

examples/ftp/src/main/java/org/apache/apex/examples/ftp/Application.java

+import com.datatorrent.api.StreamingApplication;
+import com.datatorrent.api.annotation.ApplicationAnnotation;
+import com.datatorrent.lib.io.AbstractFTPInputOperator.FTPStringInputOperator;
+


Please add some description about the application.

@chaithu14 Added description and comments.

Please add README.md for the example app similar to other example apps

amberarrow · 2017-04-30T14:13:54Z

docs/operators/ftpInputOperator.md

+(org.apache.hadoop.fs.ftp.FTPFileSystem)
+
+## Class Diagram
+![FTPInputOperator class diagram](images/ftpInputOperator/classdiagram.png)


Please show the complete class diagram including all the interfaces implemented in the base class.

@amberarrow this operator extends the AbstractFileInputOperator in which the interfaces are implemented. The FTP input operator over-rides the getFSInstance() and partitioned() method. I feel adding the AbstractFileInputOperator to the class diagram will take too much space with no addition of value which is specific to the FTP input operator. Thoughts?

amberarrow · 2017-04-30T14:14:57Z

docs/operators/ftpInputOperator.md

+=============
+
+## Operator Objective
+This operator is designed to scan a directory from an FTP server for files, read


Mention the class name here, e.g. This operator (AbstractFTPInputOperator) ...

amberarrow · 2017-04-30T14:15:34Z

docs/operators/ftpInputOperator.md

+This operator is designed to scan a directory from an FTP server for files, read
+and split file content into tuples such as lines or blocks of bytes, and finally
+emit them on the output port for further processing by downstream operators.
+The operator extends the AbstractFileInputOperator. It overrides the


Backquotes around literal class name (several places)

amberarrow · 2017-04-30T14:17:45Z

docs/operators/ftpInputOperator.md

+### Ports
+Because this is an input operator, there are no input ports.
+
+- output: output port on which data is emitted. A downstream operator receives tuples from the FTPInputOpertor


Make the first "output" bold since it is a sort of mini-title.
The second sentence is not especially useful and can be removed.

amberarrow · 2017-04-30T14:20:24Z

docs/operators/ftpInputOperator.md

+2. **source**: the directory path from where to scan and read files.
+3. **username**: the username for authenticating against the FTP server. This is
+an optional property and can be skipped when anonymous FTP is enabled
+4. **password**: the password to be used in conjunction with the above username.


Show examples of setting these properties both in code as well as in properties files (but please test your examples with an actual running application before adding them).

francisf · 2017-05-08T09:14:45Z

@amberarrow please go through the review comments.

shubham-pathak22

For operator documentation, I recommend formatting the page to have it consistent with other docs. E.g adding port and properties information in a table, code block for static and dynamic partitioning, clearly specifying operator related info like version number, package names etc. You may refer to https://github.com/apache/apex-malhar/blob/master/docs/operators/csvParserOperator.md

shubham-pathak22 · 2017-05-09T10:30:49Z

examples/ftp/src/main/java/org/apache/apex/examples/ftp/Application.java

+import com.datatorrent.api.StreamingApplication;
+import com.datatorrent.api.annotation.ApplicationAnnotation;
+import com.datatorrent.lib.io.AbstractFTPInputOperator.FTPStringInputOperator;
+


Please add README.md for the example app similar to other example apps

shubham-pathak22

Added few comments. Rest all looks good to me. Make sure commits are sqashed and jenkins checks are passing

shubham-pathak22 · 2017-05-11T09:12:48Z

docs/operators/ftpInputOperator.md

+
+#### Dynamic Partitioning
+Dynamic partitioning -- changing the number of partitions of one or more operators
+in a running application -- can be achieved in multiple ways:


Add a code block of how to use ThroughputBased / LatencyBased Partitioner as an example in addition to the description.

This is done via the setPartitionCount method which is a part of the super class AbstractFileInputOperator.

shubham-pathak22 · 2017-05-11T09:14:46Z

docs/operators/ftpInputOperator.md

+  count, set the `repartitionRequired` flag in the response.
+
+### Configuration
+


You may move this section above. Have it just after Ports section

shubham-pathak22 · 2017-05-11T10:27:41Z

examples/ftp/src/main/java/org/apache/apex/examples/ftp/Application.java

+    FTPStringInputOperator reader = dag.addOperator("Reader", new FTPStringInputOperator());
+    //Set properties for the FTP input operator
+    reader.setHost("localhost");
+    reader.setUserName("ftp");


I suggest to set these through property files instead of in Application.java ( they way it has been mentioned in the README.md )

This was done in response to earlier comment on this PR for setting properties in the example application itself. Currently properties are being set in the code as well as properties.xml

shubham-pathak22 · 2017-05-11T10:27:59Z

examples/ftp/src/main/java/org/apache/apex/examples/ftp/Application.java

+    // writer that writes strings to a file on hdfs
+    StringFileOutputOperator writer = dag.addOperator("Writer", new StringFileOutputOperator());
+    //Set properties for the output operator
+    writer.setFilePath("malhar_examples/ftp");


Set these through property files

This was done in response to earlier comment on this PR for setting properties in the example application itself. Currently properties are being set in the code as well as properties.xml

shubham-pathak22 · 2017-05-16T06:12:40Z

@francisf Looks good to me. Please make sure Travis build checks are passed. Will merge it in a day if there are no other comments from the community

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch 2 times, most recently from fd21300 to 48f0cf0 Compare April 11, 2017 08:23

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch 2 times, most recently from 291a94b to 577ba2f Compare April 21, 2017 06:29

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch from 577ba2f to 80537e3 Compare April 26, 2017 07:36

chaithu14 reviewed Apr 26, 2017

View reviewed changes

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch from 80537e3 to 795707b Compare April 27, 2017 06:41

amberarrow reviewed Apr 30, 2017

View reviewed changes

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch 2 times, most recently from b2f6ab7 to 53ae2c6 Compare May 8, 2017 09:09

shubham-pathak22 reviewed May 9, 2017

View reviewed changes

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch from 53ae2c6 to 1e916e5 Compare May 10, 2017 09:32

shubham-pathak22 reviewed May 11, 2017

View reviewed changes

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch from 1e916e5 to fab1422 Compare May 11, 2017 10:45

APEXMALHAR-2463 FTP input operator sample app, and documentation

c3f86f2

francisf force-pushed the APEXMALHAR-2463_FTP_input_example branch from fab1422 to c3f86f2 Compare May 16, 2017 06:52

asfgit merged commit c3f86f2 into apache:master May 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

APEXMALHAR-2463 FTP input operator sample app, and documentation #596

APEXMALHAR-2463 FTP input operator sample app, and documentation #596

francisf commented Mar 30, 2017

amberarrow commented Apr 25, 2017

chaithu14 left a comment

chaithu14 Apr 26, 2017

francisf Apr 27, 2017

shubham-pathak22 May 9, 2017

amberarrow Apr 30, 2017

francisf May 8, 2017

amberarrow Apr 30, 2017

francisf May 8, 2017

amberarrow Apr 30, 2017 •

edited

francisf May 8, 2017

amberarrow Apr 30, 2017

francisf May 8, 2017

amberarrow Apr 30, 2017

francisf May 8, 2017

francisf commented May 8, 2017

shubham-pathak22 left a comment

shubham-pathak22 May 9, 2017

shubham-pathak22 left a comment

shubham-pathak22 May 11, 2017

francisf May 11, 2017

shubham-pathak22 May 11, 2017

shubham-pathak22 May 11, 2017

francisf May 11, 2017

shubham-pathak22 May 11, 2017

francisf May 11, 2017

shubham-pathak22 commented May 16, 2017

		count, set the `repartitionRequired` flag in the response.

		### Configuration

APEXMALHAR-2463 FTP input operator sample app, and documentation #596

APEXMALHAR-2463 FTP input operator sample app, and documentation #596

Conversation

francisf commented Mar 30, 2017

amberarrow commented Apr 25, 2017

chaithu14 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amberarrow Apr 30, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

francisf commented May 8, 2017

shubham-pathak22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubham-pathak22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubham-pathak22 commented May 16, 2017

amberarrow Apr 30, 2017 •

edited