Fix finding topic paths for s3

[nivemaham]

Current implementation processes only the first topic listed in the objects lists for s3 implementation.
s3 client's listObject() only lists available objects. It doesn't list directories as separate item. Hence isDir is also always false.

This fix allows to have the findTopicPaths() function overridden for s3 and old implementation for other SourceStorage.

[blootsvoets]

I'm wondering if you could call s3Client.listObjects(bucket, path.toString(), false) in S3SourceStorage.list (last argument is recursive) instead of using refactoring. I see now that the default is recursive is true, meaning that there is just an object list. But setting it to false yields more of a directory structure. This avoids the potentially huge overhead of listing all files just to get the topic names.

[nivemaham]

@blootsvoets Yes. I gave it a try yesterday. It didn't work. I had a better look on it today.
It returns an infinitive loop of Item with only the first order directory under the bucket/path we are looking into. Plus if it is Item of a directory, it throws NullPointerException if we try to call item.lastModified()

    override fun list(path: Path): Sequence<SimpleFileStatus> = s3Client.listObjects(bucket, path.toString(), false)
            .asSequence()
            .map {
                val item = it.get()
                logger.info("Item is {} {}", item.objectName(), item.isDir)

                if (item.isDir) {
                    SimpleFileStatus(Paths.get(item.objectName()), item.isDir, null)
                } else {
                    SimpleFileStatus(Paths.get(item.objectName()), item.isDir, item.lastModified().toInstant())
                }

            }

The result is

2020-06-10 08:44:19 INFO  Application:187 - Starting at 2020-06-10T08:44:19.714827...
2020-06-10 08:44:22 INFO  YAMLConfigLoader:319 - Loaded config from /etc/restructure/restructure.yml
2020-06-10 08:44:23 INFO  S3TargetStorage:43 - Object storage configured with endpoint http://minio:9000/ in bucket radar-output-storage
2020-06-10 08:44:24 INFO  S3TargetStorage:49 - Bucket radar-output-storage already exists.
2020-06-10 08:44:24 INFO  RedisRemoteLockManager:16 - Managing locks as ID c8788a99-8b0f-4396-bc44-01c8ab805bbd
2020-06-10 08:44:24 INFO  Application:96 - Running as a Service with poll interval of 300 seconds
2020-06-10 08:44:24 INFO  Application:97 - Press Ctrl+C to exit...
2020-06-10 08:45:39 INFO  Application:122 - In:  /topics
2020-06-10 08:45:39 INFO  Application:123 - Out: /output
2020-06-10 08:45:39 INFO  RadarKafkaRestructure:74 - Scanning topics...
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:39 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true
2020-06-10 08:45:40 INFO  S3SourceStorage:23 - Item is /topics/ true

[blootsvoets]

Using a slash after the path name fixes the infinite recursion. Making lastModified optional fixes the nullpointerexception.

[nivemaham]

Thanks. Adding the / does the trick. Yes. I have made the LastModified optional already.

[nivemaham]

I will proceed with the merge since you have made your changes @blootsvoets .