HTTP/S?

[andrewcstewart]

Hi there. The README says this tap should work with HTTPS, but looking at the code it looks like it doesn't?

[ets]

smart_open supports http/s so it can open and read files using the protocol. However - take a look at line 101 here https://github.com/ets/tap-spreadsheets-anywhere/blob/master/tap_spreadsheets_anywhere/file_utils.py
That's the section where http/s isn't yet supported...it's where we collect all the available source files for a config block. For URLs we should probably just return a single target since a file listing isn't practical. Relatively easy PR if you're interested in contributing.

[andrewcstewart]

Yep, I can take a stab at it.

Looking at the other list_files_in_* functions, I'm guessing we'd just do a simple passthrough of the URL here, returned as a single element list?

[ets]

Agreed - that seems like the best fit for a URL.

[andrewcstewart]

Working on it now. Does the list_files function need to do anything else other than this?

def list_files_in_http_bucket(uri, search_prefix=None):
    entries = [uri]
    LOGGER.info("Found {} files.".format(entries))
    return entries

[ets]

Added support in the latest update.