Google Sheets Query support in sql processor

[harshdoesdev]

Background

We are adding support for Google Sheets Query to the sql processor. You can find more information on this and participate in the related discussion.

How it Works

The Google Charts API allows you to query a Google Sheet much like you would with an SQL database, though it currently only supports data retrieval queries such as "SELECT." Our primary objective is to enable users to provide the URL of their Google Sheet as the db parameter for the sql processor. Currently, only public sheets are supported, but we plan to add support for private sheets in the future.

Example:

-- import: fastn/processors as pr

-- record user:
string Name:
integer Age:
string Gender:

-- user list users:
$processor$: pr.sql
db: https://docs.google.com/spreadsheets/d/1FOSZWtfQni0sNLLzbyhHGqzpZcRJx0_TLO8UGtnDXiY
sheet: Sheet1

SELECT *

-- ftd.text: $u.Name
for: $u in $users

Todos

• Basic Processor Setup
• Google Charts API Integration
• Data type conversion
• Getting a basic demo running
• Resolve Named parameters
• Move from CSV to JSON
• Handle Formatted values
• Query specific sheet by passing sheet name as sheet
• Documentation

Updates

• For now I am going ahead with sheet and db instead of $sheet$ and $db$, we will deprecate these in a follow up PR and replace these with $sheet$and$db$`.
• Skipping the fbt tests for now, will do manual testing instead. will add fbt tests in a follow up PR

[amitu]

@Arpita-Jaiswal can you take a look at the fastn-core/src/library2022/processor/google_sheets.rs, there is a lot of stuff happening there and I want your opinion.

[harshdoesdev]

@amitu I have temporarily removed the timestamp conversion for now. also, I have added a new optional processor argument named sheet where the user can specify the name of the sheet they want to query.

Example:

-- user list users:
$processor$: pr.sql
db: https://docs.google.com/spreadsheets/d/1FOSZWtfQni0sNLLzbyhHGqzpZcRJx0_TLO8UGtnDXiY
sheet: Users
name: John Doe

SELECT * WHERE A = $name::STRING

[amitu]

The sheet argument conflicts if the sheet is a column name. Should we name processor arguments as $<arg-name>$, eg $sheet$: <sheet name> and same with db.

[harshdoesdev]

The sheet argument conflicts if the sheet is a column name. Should we name processor arguments as $<arg-name>$, eg $sheet$: <sheet name> and same with db.

I agree, but it should not conflict with the column name because these words are rarely used, and this looks cleaner without '$'. However, this is just my opinion.

[amitu]

It looks cleaner, but if conflict happens user will have no choice. sheet and db are both common enough terms. Let's switch to $foo$ for all processor arguments as this is a general problem and we need a general / consistent solution.

[harshdoesdev]

It looks cleaner, but if conflict happens user will have no choice. sheet and db are both common enough terms. Let's switch to $foo$ for all processor arguments as this is a general problem and we need a general / consistent solution.

Yes, although these argument names need not be the same as the column name, many users choose to name them to match the column name. Therefore, I agree that it can lead to conflicts.