Source File: add support for directories

[eugene-kulak]

Tell us about the problem you're trying to solve

Currently, we can point URL to file only, so we always have a single stream. It would be nice to list files in the directory in case URL points to a directory object.

Describe the solution you’d like

Given file structure

files_folder/
   file1.csv
   file2.xls
   file3.csv
   inner_folder/
      file4.csv

For url="ssh://127.0.0.1:200/files_folder/" and format=csv we should see following streams

streams:
   file1  
   file3
   inner_folder_file4.csv

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

┆Issue is synchronized with this Asana task by Unito

[ChristopheDuong]

• A folder could be considered as a Database Schema
• we could introduce a regex pattern to match multiple files that would form the same stream (they all need to be the same schema, maybe not the same file format?) => that way you can upload a large file in different chunk/partitionned files
• you can have different streams in a folder as we do with other sources

But then comes the question of incremental refreshes with such a source...
Maybe the state should keep track of filenames, modification dates, checksum, etc from replicated files?)

[eugene-kulak]

@ChristopheDuong

• I like this analogy
• To support this we need somehow define a rule to extract base stream name from file name. Then we can apply a regex to all filenames and find all partitions for each stream.
• Regarding different file formats, I would allow only compressed variations, i.e csv and csv.gz.
• Yes we can, the problem/complication that I see is that we have too many stream-specific options on the connector level (format, reader_options).

[sherifnada]

this is happening in the S3 connector, and won't happen in the file source per say. So closing as wontfix.