-
-
Notifications
You must be signed in to change notification settings - Fork 380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes issue #88 Added a 'parents' functionality #169
Conversation
…art_open into add-parent-option
@@ -36,6 +36,8 @@ | |||
from ssl import SSLError | |||
import sys | |||
|
|||
import errno | |||
from ntpath import split |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you use ntpath instead of os.path?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @melvin15may for PR! Can you describe, what's pros/cons of this feature? |
This PR adds a feature which basically helps segregate files automatically instead of creating directory structure beforehand and then using those. Especially useful when streaming to a new file. |
@melvin15may What are the advantages/disadvantages of this PR over the below alternative? def my_open(path, *args, **kwargs): # untested pseudo-code
subdir = os.path.dirname(path)
if not os.path.isdir(subdir):
os.mkdir(subdir)
return smart_open.smart_open(path, *args, **kwargs) and then use |
Gives the user an option ( |
@melvin15may In my opinion, this pull request and the On the other hand, including this PR in the library code has several disadvantages that I can think of. In no particular order:
@menshikh-iv Given that the advantages of merging this PR are smaller than the disadvantages, my opinion is that we should handle directory creation outside the library. |
I agree with you @mpenkov, looks simple, but in depth can be slightly painful for us. Wdyt @piskvorky? |
Up to you guys. I personally never needed such feature, it sounds dangerous to me to be creating deep directories transparently (I'd prefer to do it explicitly in upstream app code, with clear comments, at a place where it's needed). But I don't know all use cases, it may be useful to others. |
@melvin15may any additional reasons? |
Ok, I close this PR. |
IMO taking on those 'disadvantages' are exactly what a utility-library does, to avoid duplication/less-optimal-reimplementations elsewhere, and thus amortize the cost of "doing it well" over all the projects that can then rely on the library. This facility is useful in cmd-line And for perhaps the most-common backend other than local volumes, S3 (& similar APIs), it's essentially a no-op – but making |
I know this is an old thread, but I agree with @gojomo - in order to transparently write to both S3 and a local file, I'm going to work around this (poorly) by checking whether the URL starts with It's a very kludgey solution that essentially undermines the cross-backend advantages of |
Second, @gojomo and @kenahoo … we're facing this same problem with SFTP. It would've been nice if smart_open provided a common path for the supported platforms without requiring addl. code. In the case of SFTP this is a bit hard, as sftpclient.open simply fails with an exception. Then we have to construct the connection (which smart_open does beautifully, but hidden internally), and create all the dirs before calling it again. |
@mpenkov wrote:
IMO the large disadvantage is that it negates any advantage of using |
@kenahoo another (admittedly unclean) approach would be that, if the dir doesn't exist then throw an exception with the underlying client (sftpclient, etc) provided within the exception. Then, we have a way of getting the client, creating the dir and pushing the file. Again, I realise this is totally unclean, but at least we can do something with it. Further, the S3 client doesn't complain at all. Isn't this a leaky abstraction in the sense that, one destination works, but another doesn't? |
Added functionality to create directory if
parents
option is set toTrue
.Similar to
mkdir -p
functionality.New PR fixing unrelated file change issues in closed PR #126