_default_delete_dir_handler throws error when using default handler

[manjuransari-zz]

We have option to overwrite the method _default_delete_dir_handler. But when we don't overwrite the method, it throws error when used. As we already have fsspec integrated, we can extend fsspec to enable deletion of files.

[selitvin]

Not sure I understand. What is the ask here?

[manjuransari-zz]

We can add our own delete handler. But when we use default handler, as above, the else code fails. We are using fsspec file system but not passing url that is acceptable by fsspec.

For example, we pass dataset_url = 'abfs://synapsemlfs/temp/file.csv'
parsed becomes ParseResult(scheme='abfs', netloc='synapsemlfs', path='/temp/file.csv', params='', query='', fragment='')

When we check fs.exist(parsed.path), we are actually checking for the url = '/temp/file.csv'. This url does not have container/bucket information. The url should be (parsed.netloc + parsed.path) to work flawlessly with fsspec. Also, fsspec support urls with storage_options within url. That particular case is also not handled.

Plus we also have support for local filesystem in fsspec, hence we would have straight implementation for both.

The acceptable code should be as follow:

from fsspec.core import strip_protocol

def _default_delete_dir_handler(dataset_url):
    resolver = FilesystemResolver(dataset_url)
    fs = resolver.filesystem()
    _dataset_url = strip_protocol(dataset_url)
    if fs.exists(_dataset_url ):
       fs.delete(_dataset_url , recursive=True)

[selitvin]

Would you like to create a PR for this?

[manjuransari-zz]

yes, I am working on it. Will raise soon.

[manjuransari-zz]

@selitvin is there any plans for a release anytime soon?

[selitvin]

Can release tomorrow.

…

On Thu, Sep 2, 2021, 11:03 PM Manjur Ansari ***@***.***> wrote: @selitvin <https://github.com/selitvin> is there any plans for a release anytime soon? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#716 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZIEDFFWNDDGUG64MBPM2DUABQJPANCNFSM5COVQ2KA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[selitvin]

petastorm==0.11.3rc0 is out

[selitvin]

0.11.3 released