FTP Connector #3

juanjovazquez · 2016-10-20T10:20:43Z

Please, check if this implementation might be enough for a first version of the connector. My team is already using it on a real project so this will be tested from the beginning.

closes #4

/cc @patriknw @ktoso @2m

patriknw

I think this is looking good. Let us know if there is something specific you would like us to take an extra look at. Then adding javadsl and Paradox.

patriknw · 2016-11-16T13:18:20Z

ftp/src/main/scala/akka/stream/alpakka/ftp/Ftp.scala

+import scala.util.Try
+
+/**
+ * @author Juan José Vázquez Delgado


please remove, explained in https://github.com/akka/alpakka/blob/master/CONTRIBUTING.md#pull-request-requirements

patriknw · 2016-11-16T13:22:55Z

ftp/src/main/scala/akka/stream/alpakka/ftp/Ftp.scala

+  def connect(
+    connectionSettings: FtpConnectionSettings
+  )(implicit ftpClient: FTPClient): Try[Handler] = Try {
+    if (ftpClient != null) {


why those null checks? isn't that a program error?

patriknw · 2016-11-16T13:29:24Z

ftp/src/main/scala/akka/stream/alpakka/ftp/FtpsSource.scala

+/*
+ * Copyright (C) 2016 Lightbend Inc. <http://www.lightbend.com>
+ */
+package akka.stream.alpakka.ftp


I think we have refactored the package conventions after you started this. We now use scaladsl and javadsl. See the other connectors in master.

patriknw · 2016-11-16T13:46:13Z

ftp/src/main/scala/akka/stream/alpakka/ftp/SFtp.scala

+/**
+ * @author Juan José Vázquez Delgado
+ */
+trait SFtp {


think about what classes should be public and what is internal. Keep the public API as small as possible. Mark internal things with private[alpakka], perhaps move to an impl package also.

juanjovazquez · 2016-11-16T17:43:28Z

@patriknw Still working on this. I plan to cut a first version this week with at least the basic operations for listing and retrieving files as streams. After that, it could be a good idea going to merge and let others to contribute for improving it. FTP is an old pal and has a lot of nuances and different use cases. Thanks for your comments!.

juanjovazquez · 2016-11-21T19:06:13Z

ftp/src/main/scala/akka/stream/alpakka/ftp/javadsl/FtpApi.scala

+      connectionSettings: RemoteFileSettings
+  ): Source[FtpFile, CompletionStage[java.lang.Long]] = {
+    import scala.compat.java8.FutureConverters._
+    implicit val ec = scala.compat.java8.FutureConverters.globalExecutionContext // TODO


@patriknw Any idea about how could I avoid this execution context import?. I need to map the Scala Future[Long] to the Java CompletionStage<java.lang.Long> and the longs conversion is asking me for an ExecutionContext. Should I use a wrapper as in IOResult?.

GraphStage has materializer.executionContext

2m · 2016-11-25T12:08:47Z

build.sbt

@@ -44,6 +44,15 @@ lazy val s3 = project
    Dependencies.S3
  )

+lazy val ftp = project
+  .in(file("ftp"))


This is not necessary, when name of the module is the same as the directory.

2m

Just a couple of nitpicks in the docs. Otherwise LGTM.

2m · 2016-11-25T12:09:19Z

docs/src/main/paradox/ftp.md

+sbt
+:   @@@vars
+    ```scala
+    libraryDependencies += "com.typesafe.akka" %% "akka-stream-alpakka-ftp" % "$version$"


s/typesafe/lightbend

2m · 2016-11-25T12:09:29Z

docs/src/main/paradox/ftp.md

+:   @@@vars
+    ```xml
+    <dependency>
+      <groupId>com.typesafe.akka</groupId>


s/typesafe/lightbend

2m · 2016-11-25T12:09:37Z

docs/src/main/paradox/ftp.md

+:   @@@vars
+    ```gradle
+    dependencies {
+      compile group: "com.typesafe.akka", name: "akka-stream-alpakka-ftp_$scala.binaryVersion$", version: "$version$"


s/typesafe/lightbend

2m · 2016-11-25T12:10:23Z

docs/src/main/paradox/ftp.md

+
+### Configuring the connection settings
+
+In order to establish a connection with the remote server, you need to provide an specialized version of a @scaladoc[RemoteFileSettings](akka.stream.alpakka.ftp.RemoteFileSettings) instance. It's specialized as it depends on the kind of server you're connecting to: FTP, FTPs or SFTP.


... you need to provide an specialized ...

2m · 2016-11-25T12:11:45Z

docs/src/main/paradox/ftp.md

+Java
+: @@snip (../../../../ftp/src/test/java/akka/stream/alpakka/ftp/FtpSourceTest.java) { #create-settings }
+
+The configuration above will create an anonymous connection with a remote FTP server in passive mode. For both FTPs and SFTP servers, you will need to provide the specialized versions of these settings: @scaladoc[FtpsSettings](akka.stream.alpakka.ftp.RemoteFileSettings.FtpsSettings) or @scaladoc[SFtpSettings](akka.stream.alpakka.ftp.RemoteFileSettings.SFtpSettings) 


This does not properly link. Use akka.stream.alpakka.ftp.RemoteFileSettings$$FtpsSettings and akka.stream.alpakka.ftp.RemoteFileSettings$$SFtpSettings

2m · 2016-11-25T12:12:19Z

docs/src/main/paradox/ftp.md

+The configuration above will create an anonymous connection with a remote FTP server in passive mode. For both FTPs and SFTP servers, you will need to provide the specialized versions of these settings: @scaladoc[FtpsSettings](akka.stream.alpakka.ftp.RemoteFileSettings.FtpsSettings) or @scaladoc[SFtpSettings](akka.stream.alpakka.ftp.RemoteFileSettings.SFtpSettings) 
+respectively.
+
+For non-anonymous connection, please provide an instance of [NonAnonFtpCredentials](akka.stream.alpakka.ftp.FtpCredentials.NonAnonFtpCredentials) instead.


Missing @scaladoc before [NonAnonFtpC...

johanandren

Looks promising, I had a few questions/opinions I have added.

johanandren · 2016-11-25T12:55:06Z

ftp/src/main/scala/akka/stream/alpakka/ftp/impl/FtpBrowserGraphStage.scala

+import scala.concurrent.{ Future, Promise }
+
+private[ftp] trait FtpBrowserGraphStage[FtpClient]
+    extends GraphStageWithMaterializedValue[SourceShape[FtpFile], Future[Done]] {


if the stage just fails for errors, and completes the same thing can be achieved by using ftpSource.watchTermination(), is there some special circumstance that the Future[Done] materialized value is needed for?

Not really. The only purpose of Future[Done] is to check termination. Should I change it and not materialize?.

Future[Done] is great for sinks, but not really needed for sources

Perfect. I'll change this.

johanandren · 2016-11-25T13:30:30Z

ftp/src/main/scala/akka/stream/alpakka/ftp/impl/FtpBrowserGraphStage.scala

+  val ftpLike: FtpLike[FtpClient]
+
+  val shape = SourceShape(Outlet[FtpFile](s"$name.out"))
+


as the blocking logic is done already in this trait I think the IODispatcher should be specified using initialAttributes here rather than when creating the flows in FtpApi (and possibly miss out on it)

johanandren · 2016-11-25T13:32:09Z

ftp/src/main/scala/akka/stream/alpakka/ftp/impl/FtpBrowserGraphStage.scala

+
+    val logic = new FtpGraphStageLogic[FtpFile, FtpClient](shape, ftpLike, connectionSettings) {
+
+      private var currentFiles: Stream[FtpFile] = Stream.empty[FtpFile]


I seem to recall that Stream leaks memory because of memoization, are you sure this is not happening here (keeping each file in memory even after emitting it downstream)

Please, take into account that the propagated element here is an FtpFile that is a mere descriptor for the physical file. The physical file is got through other Source, the FtpSourceIOGraphStage in the form of stream of ByteString. So, IMHO, the scala.collection.immutable.Stream should not be a problem here, right?

If you list a directory with 50 000 files, it would (if I'm right about memoization) keep all of them in memory until the stream completes which isn't very nice.

Wow!. I'll check this right away. Thanks.

Finally changed the Stream for a strict sequence as I'm quite sure that we had memoization as you mentioned. Thanks!.

johanandren · 2016-11-25T13:36:37Z

ftp/src/main/scala/akka/stream/alpakka/ftp/impl/FtpSourceFactory.scala

+  protected val ftpIOSourceName: String = FtpsIOSourceName
+}
+
+private[ftp] trait SFtpSource { _: FtpSourceFactory[_] =>


if FTP is cased as Ftp shouldn't SFTP be Sftp rather than SFtp?

It makes sense, but I find ugly Sftp personally. I suppose this is a matter of taste. Other opinions?.

I don't care super much about which one of them, it was more about consistency, both should follow the same pattern, no?

Accepted. I changed the naming. Thanks!

juanjovazquez · 2016-11-25T15:09:49Z

ftp/src/main/scala/akka/stream/alpakka/ftp/impl/FtpBrowserGraphStage.scala

+
+  override def initialAttributes: Attributes =
+    super.initialAttributes and Attributes.name(name) and IODispatcher
+


@johanandren Please, can you confirm that this is what you mean regarding initial attributes?

johanandren

LGTM

johanandren · 2016-11-29T09:32:49Z

Thanks @juanjovazquez !

ktoso mentioned this pull request Oct 21, 2016

FTP connector #4

Closed

juanjovazquez force-pushed the wip-ftp-support branch 2 times, most recently from 57d8658 to 76649d3 Compare October 28, 2016 07:42

juanjovazquez force-pushed the wip-ftp-support branch from 83f30f0 to b803a05 Compare November 11, 2016 16:13

patriknw reviewed Nov 16, 2016

View reviewed changes

juanjovazquez force-pushed the wip-ftp-support branch from 566b742 to 7c0ddb0 Compare November 18, 2016 18:59

juanjovazquez commented Nov 21, 2016

View reviewed changes

juanjovazquez force-pushed the wip-ftp-support branch from e948be9 to cb22209 Compare November 22, 2016 17:55

juanjovazquez changed the title ~~WIP: FTP support~~ FTP Connector Nov 22, 2016

juanjovazquez force-pushed the wip-ftp-support branch from cb22209 to e99d974 Compare November 25, 2016 11:06

2m reviewed Nov 25, 2016

View reviewed changes

2m suggested changes Nov 25, 2016

View reviewed changes

johanandren reviewed Nov 25, 2016

View reviewed changes

juanjovazquez force-pushed the wip-ftp-support branch 2 times, most recently from 1f664ca to 29143af Compare November 25, 2016 15:39

juanjovazquez commented Nov 25, 2016

View reviewed changes

juanjovazquez force-pushed the wip-ftp-support branch from 29143af to 0895b04 Compare November 25, 2016 18:33

FTP Connector (closes akka#4)

443f00d

juanjovazquez force-pushed the wip-ftp-support branch from 0895b04 to 443f00d Compare November 25, 2016 18:53

johanandren approved these changes Nov 28, 2016

View reviewed changes

2m approved these changes Nov 29, 2016

View reviewed changes

johanandren merged commit a2a20e8 into akka:master Nov 29, 2016

juanjovazquez deleted the wip-ftp-support branch November 29, 2016 10:02

longshorej mentioned this pull request Feb 12, 2019

mqtt-streaming: Prefer wrapping instead of reissuing packet ids #1489

Merged


		### Configuring the connection settings

		In order to establish a connection with the remote server, you need to provide an specialized version of a @scaladoc[RemoteFileSettings](akka.stream.alpakka.ftp.RemoteFileSettings) instance. It's specialized as it depends on the kind of server you're connecting to: FTP, FTPs or SFTP.

		val ftpLike: FtpLike[FtpClient]

		val shape = SourceShape(Outlet[FtpFile](s"$name.out"))


		val logic = new FtpGraphStageLogic[FtpFile, FtpClient](shape, ftpLike, connectionSettings) {

		private var currentFiles: Stream[FtpFile] = Stream.empty[FtpFile]


		override def initialAttributes: Attributes =
		super.initialAttributes and Attributes.name(name) and IODispatcher

FTP Connector #3

FTP Connector #3

Conversation

juanjovazquez commented Oct 20, 2016 • edited Loading

patriknw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanjovazquez commented Nov 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanandren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanjovazquez Nov 25, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanandren left a comment

Choose a reason for hiding this comment

johanandren commented Nov 29, 2016

juanjovazquez commented Oct 20, 2016 •

edited

Loading

juanjovazquez Nov 25, 2016 •

edited

Loading