Strange folder structure

[lukemarsden]

when using dataset.download(), the files gets downloaded into a bonkers directory structure like http%blah

[lukemarsden]

deprio this because we're demoing via tabular datastores which go to a pandas dataframe

[lukemarsden]

reprio becuase we want to support files and tables now

[albscui]

According to Andrei, the download path is not handled by rslex currently. This might be resolved automatically by a future update.

[lukemarsden]

I wonder if we control the path that gets passed back to whatever code handles the downloads though

…

On Tue, 6 Jul 2021, 19:22 Albert, ***@***.***> wrote: According to Andrei, the download path is not handled by rslex currently. This might be resolved automatically by a future update. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACATUSILUSYQQAYDIBYDALTWNCYFANCNFSM4637W7WQ> .

[anliakho2]

This is caused by the inability to detect folder structure from python (because it does a list today and tries to find a common prefix)
There needs to be an improvement in the SDK to make the logic smarter
As of right now it does the safe option of creating a fully qualified path so even if your dataset has multiple paths pointing to different pachyderm clusters files dowloaded would not collide.

[albscui]

I wonder if we control the path that gets passed back to whatever code handles the downloads though
…
On Tue, 6 Jul 2021, 19:22 Albert, @.***> wrote: According to Andrei, the download path is not handled by rslex currently. This might be resolved automatically by a future update. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACATUSILUSYQQAYDIBYDALTWNCYFANCNFSM4637W7WQ .

As a matter of fact, we do! We were adding the http://localhost:30600... prefix to the StreamInfos returned by the Searcher. We didn't need to do this because the rslex-http-stream library calls the ReadRequest API which is implemented by the RequestBuilder. The RequestBuilder contains the HTTP schema, host, and port info already.