Add read_text and read_blob table functions

[Maxxen]

This PR adds a set of new table functions, read_text and read_blob that accepts a file name, list of file names or glob pattern, and outputs the content of each file as a varchar or blob, as well as additional metadata such as the file size and last modified time.

In read_blob the file content is returned as BLOB, but in read_text the file content is first validated to be valid UTF-8 and then returned as VARCHAR. If read_text attempts to read a file with invalid UTF-8 an error is thrown suggesting to use read_blob instead.

Example:

 SELECT size, parse_path(filename), content  FROM read_text('test/sql/table_function/files/*.txt');
┌───────┬───────────────────────────────────────────────┬──────────────┐
│ size  │             parse_path(filename)              │   content    │
│ int64 │                   varchar[]                   │   varchar    │
├───────┼───────────────────────────────────────────────┼──────────────┤
│    12 │ [test, sql, table_function, files, one.txt]   │ Hello World! │
│     2 │ [test, sql, table_function, files, three.txt] │ 42           │
│    10 │ [test, sql, table_function, files, two.txt]   │ Föö Bär      │
└───────┴───────────────────────────────────────────────┴──────────────┘

SELECT size, content, filename  FROM read_blob('test/sql/table_function/files/*');
┌───────┬──────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────┐
│ size  │                                             content                                              │                filename                 │
│ int64 │                                               blob                                               │                 varchar                 │
├───────┼──────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────┤
│   178 │ PK\x03\x04\x0A\x00\x00\x00\x00\x00\xACi=X\x14t\xCE\xC7\x0A\x00\x00\x00\x0A\x00\x00\x00\x09\x00…  │ test/sql/table_function/files/four.blob │
│    12 │ Hello World!                                                                                     │ test/sql/table_function/files/one.txt   │
│     2 │ 42                                                                                               │ test/sql/table_function/files/three.txt │
│    10 │ F\xC3\xB6\xC3\xB6 B\xC3\xA4r                                                                     │ test/sql/table_function/files/two.txt   │
└───────┴──────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────┘

In cases where the underlying filesystem is unable to provide some of this data due to throwing a NotImplementedException (or a conversion error in the case of the last_modified column, because HTTPFS can't always return a valid timestamp), the cell is set to NULL instead.

This table function also utilizes projection pushdown so as to not compute properties unnecessarily. So you could e.g. use this to glob a directory full of huge files to get the file size in the size column, as long as you omit the content column the data wont be read into DuckDB.

[Tishj]

I feel for most purposes this would be used, VARCHAR is expected, maybe we can add an alternative read_file_utf8 to support this?
Or even, since VARCHAR is likely the most commonly desired output, have read_file be VARCHAR and have read_file_bytes output BLOB?

As far as I know converting BLOB to VARCHAR is cumbersome.

[Maxxen]

I feel for most purposes this would be used, VARCHAR is expected, maybe we can add an alternative read_file_utf8 to support this? Or even, since VARCHAR is likely the most commonly desired output, have read_file be VARCHAR and have read_file_bytes output BLOB?

As far as I know converting BLOB to VARCHAR is cumbersome.

Hmm, I don't think adding a decode call is that bad, but I am sympathetic to this. Maybe we could add a read_text table macro that performs the decode on the data column

[Tishj]

I forgot about decode, that's not too bad actually, but read_text sounds nice

[carlopi]

I tried to use it (and break it), looks to be working nicely, and last_modified can be handy.

Only concern I have is that read_* function names are becoming a scarce resource, and it would be cool to eventually have a read_XY(['path/to/fileA', 'path/to/fileB']) that automagically detect the common type and act on that.
But I think read_file is what this function is doing, so probably it's good. Plus there is a sqlite CLI functions that is called readfile and gets the blob out of a file, so it does match nicely.

[Maxxen]

I was gonna rewrite it read_blob, and then push an alias that tries to decode as text called read_text. Thoughts?

[Mytherin]

Thanks for the PR! Looks good - some comments:

[Maxxen]

@Mytherin Resolved your comments - having to deal with the rowid column (as I suppose is the case in with COUNT(*)) when enabling projection pushdown was a new for me! Will have to remember to add that in some of the extensions I work on :)

[Mytherin]

Thanks!