Skip to content
Permalink
Browse files

Clean up per review comments

  • Loading branch information...
dturanski committed Nov 5, 2018
1 parent 4a7b800 commit f2b2f1f34dc89b9dbee1a9b5f9072a574a986ad1
@@ -32,6 +32,7 @@ The jar is published to the https://repo.spring.io/libs-snapshot-local/io/sprin
Now we create a remote directory on the SFTP server and a local directory where the batch job expects to find files.
+
NOTE: If you are using a remote SFTP server, create the remote directory on the SFTP server.
Since we are using the local machine as the SFTP server, we will create both the local and remote directories on the local machine.
+
```
$ mkdir -p /tmp/remote-files /tmp/local-files
@@ -42,7 +43,7 @@ $ mkdir -p /tmp/remote-files /tmp/local-files
+
With our Spring Cloud Data Flow server running, we register the `sftp-dataflow` source and `task-launcher-dataflow` sink.
The `sftp-dataflow` source application will do the work of polling the remote directory for new files and downloading them to the local directory.
As each file is received, it emits a message for the `task-launcher-dataflow` sink to launch the task for that file.
As each file is received, it emits a message for the `task-launcher-dataflow` sink to launch the task to process the data from that file.
+
In the Spring Cloud Data Flow shell:
+
@@ -54,7 +55,7 @@ dataflow:>app register --name task-launcher --type sink --uri maven://org.spring
Successfully registered application 'sink:task-launcher'
----
+
. Register and create the file ingest task. If using the published jar set `--uri maven://io.spring.cloud.dataflow.ingest:ingest:1.0.0.BUILD-SNAPSHOT`:
. Register and create the file ingest task. If you're using the published jar, set `--uri maven://io.spring.cloud.dataflow.ingest:ingest:1.0.0.BUILD-SNAPSHOT`:
[source,console,options=nowrap]
dataflow:>app register --name fileIngest --type task --uri file:///path/to/target/ingest-X.X.X.jar
Successfully registered application 'task:fileIngest'
@@ -63,7 +64,8 @@ Created new task 'fileIngestTask'
+
. Create and deploy the stream
+
Now lets create and deploy the stream which will start polling the SFTP server and when new files arrive launch the batch job.
Now lets create and deploy the stream.
Once deployed, the stream will start polling the SFTP server and, when new files arrive, launch the batch job.
+
NOTE: Replace `<user>` and '<pass>` below.
The `<username>` and `<password>` values are the credentials for the local (or remote) user.
@@ -182,7 +182,7 @@ dataflow>app register --name sftp --type source --uri file:<project-directory>/s
Follow the instructions for building and running the main SFTP File Ingest demo, for your preferred platform, up to the `Add Data Step`.
If you have already completed the main exercise, restore the data to its initial state, and redeploy the stream:
+
* Clean the data directories
* Clean the data directories (e.g., `tmp/local-files` and `tmp/remote-files`)
* Execute the SQL command `DROP TABLE PEOPLE;` in the database
* Undeploy the stream, and deploy it again to run the updated `sftp` source
+
@@ -224,7 +224,7 @@ If we view the `PEOPLE` table, it should look something like this:
+
image::people_table_1.png[title="People Data"]
+
Now let'supdate the remote file, using SFTP `put` or if using the local machine as an SFTP server:
Now let's update the remote file, using SFTP `put` or if using the local machine as an SFTP server:
+
```
$touch /tmp/remote-files/names_aa.csv
@@ -3,9 +3,9 @@

===== Additional Prerequisites

NOTE: Running this demo in Cloud Foundry requires the a shared file system that is accessed by apps running in different containers.
NOTE: Running this demo in Cloud Foundry requires a shared file system that is accessed by apps running in different containers.
This feature is provided by https://docs.pivotal.io/pivotalcf/2-3/devguide/services/using-vol-services.html[NFS Volume Services].
Using Volume Services with SCDF requires that we provide `nfs` configuration via `cf create-service` rather than `cf bind-service`.
To use Volume Services with SCDF, it is required that we provide `nfs` configuration via `cf create-service` rather than `cf bind-service`.
Cloud Foundry introduced the `cf create-service` configuration option for Volume Services in version 2.3.

* A Cloud Foundry instance v2.3+ with NFS Volume Services https://docs.pivotal.io/pivotalcf/2-3/opsguide/enable-vol-services.html[enabled]
@@ -67,7 +67,7 @@ $ sudo chown <uid>:<gid> /export/shared-files
+
With our Spring Cloud Data Flow server running, we register the `sftp-dataflow` source and `task-launcher-dataflow` sink.
The `sftp-dataflow` source application will do the work of polling the remote directory for new files and downloading them to the local directory.
As each file is received, it emits a message for the `task-launcher-dataflow` sink to launch the task for that file.
As each file is received, it emits a message for the `task-launcher-dataflow` sink to launch the task to process the data from that file.
+
In the Spring Cloud Data Flow shell:
+
@@ -88,7 +88,8 @@ Created new task 'fileIngestTask'
+
. Create and deploy the stream
+
Now lets create and deploy the stream which will start polling the SFTP server and when new files arrive launch the batch job.
Now lets create and deploy the stream.
Once deployed, the stream will start polling the SFTP server and, when new files arrive, launch the batch job.
+
NOTE: Replace `<user>`, '<pass>`, and `<host>` below.
The `<host>` is the SFTP server host, `<user>` and `<password>` values are the credentials for the remote user.

0 comments on commit f2b2f1f

Please sign in to comment.
You can’t perform that action at this time.