Merge pull request #22 from centerofci/views_spec

Product spec for Views

.gitignore

@@ -1 +1,30 @@
 *.pyc
+
+# Don't commit Obsidian files, allows wiki to be edited in Obsidian.
+.obsidian
+
+## MAC ##
+
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk

product/specs.md

@@ -0,0 +1,11 @@
+---
+title: Product Specs
+description: List of product specs
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+- [Views](/product/specs/2022-01-views)

product/specs/2022-01-views.md

@@ -0,0 +1,31 @@
+---
+title: Views in Mathesar
+description: Spec for Views in Mathesar
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+**Date**: 2022-01
+**Author**: Kriti Godey
+
+# Introduction
+
+Fundamentally, **Views** are saved database queries. This means that in order to work with Views in Mathesar, we need to translate every concept that can be used in [PostgreSQL queries](https://www.postgresql.org/docs/14/queries.html) to our end users in a user-friendly way.
+
+Since this spec will be long, I've decided to split it up into multiple pages.
+
+- [Assumptions](/product/specs/2022-01-views/01-assumptions)
+- [Modeling View Data](/product/specs/2022-01-views/02-modeling-view-data)
+- [Modeling the View Query](/product/specs/2022-01-views/03-modeling-view-query)
+- [Modeling View Columns](/product/specs/2022-01-views/04-modeling-view-columns)
+- [UI Requirements for Views](/product/specs/2022-01-views/05-ui-requirements-for-views)
+- [Mapping DB Queries to Views](/product/specs/2022-01-views/06-mapping-db-queries-to-views)
+
+There are still some details to be figured out, including:
+- which database functions we'll support for the alpha release and how that will work.
+- how we'll handle different table `JOIN` types.
+
+This spec will be updated with more details on those and other questions raised once the initial concepts listed here have been ironed out.

product/specs/2022-01-views/01-assumptions.md

@@ -0,0 +1,17 @@
+---
+title: Views: Assumptions
+description: 
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+## Assumptions
+I'm making the following assumptions in the rest of the spec about how we want to work with Views in Mathesar.
+
+- We **do not** need to support creating or editing Views based on every conceivable database query in Mathesar. We will be focusing on allowing common use cases.
+- We **do** need to support viewing Views based on any conceivable database query correctly, even if they can't be edited. Users should be able to connect a database with existing Views to Mathesar and have those Views show up correctly.
+- At the moment, we **only** care about the final output of the views. If a view uses a subquery, CTE, union, intersection, etc. internally, we will not be representing those to the user in the UI (unless they look at the underlying SQL query).
+- We eventually want to allow users to create Views using SQL queries through the web interface, but this will not be prioritized for the alpha release.

product/specs/2022-01-views/02-modeling-view-data.md

@@ -0,0 +1,33 @@
+---
+title: Views: Modeling View Data
+description:
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+Here's how I think we should model views in our backend and API. Each heading represents an attribute of Views.
+
+## Query
+This is the SQL query that defines the view. This is **required**. 
+
+Query attributes are defined [in a separate page](/product/specs/2022-01-views/03-modeling-view-query).
+
+## Columns
+Output columns of the View's Query, they will be shown in the UI.  At least one column is **required** for a view.
+
+Column attributes are defined [in a separate page](/product/specs/2022-01-views/04-modeling-view-columns).
+
+## Rows
+These are the output rows of the View's Query. Rows are **not required**, Views can contain 0 rows.
+
+## Filters
+Views can have filters applied to the data in the View. This is similar to Table filters.
+
+## Sorting
+Views can have sorting applied to the data that's present in the View. This is similar to Table sorting.
+
+## Groups
+Groups are similar to table grouping, they sort and then visually group rows into sections.

product/specs/2022-01-views/03-modeling-view-query.md

@@ -0,0 +1,69 @@
+---
+title: Views: Modeling the View Query
+description:
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+In some cases, we will be able to break down a View's Query into simpler parts that the user can edit. Here's how I think we should model those simpler parts in our API and UI. Each heading represents an attribute of a View Query.
+
+Here's an example table named `Movies` that I'll use to illustrate the attributes below.
+
+| ID | Title | Year | Genre |
+|-|-|-|-|
+| 1 | The River Wild | 1994 | Thriller |
+| 2 | Don't Look Up | 2021 | Comedy |
+| 3 | Daylight | 1996 | Action |
+| 4 | Jason Bourne | 2016 | Action |
+
+## Editable
+A View Query is **editable** if we can break it down into the simpler parts described below. Otherwise, the View Query cannot be edited and we won't have any filters, sorts, or aggregations to show.
+
+## Filters
+A View Query can have filters applied. These filters are not necessarily related to the columns that are present in the view.
+
+Using the example table above, imagine a view created from the query `SELECT ID, Title FROM Movies WHERE Year > 2000;` This will return this view: which is filtered by Year even though it's not a column in the View.
+
+| ID | Title |
+|-|-|
+| 2 | Don't Look Up |
+| 4 | Jason Bourne |
+
+## Sorting
+Views can have sorting applied. These sorts are not necessarily related to the columns that are present in the view.
+
+Using the example table above, imagine a view created from the query `SELECT ID, Title FROM Movies ORDER BY Year;` This will return this view, which is ordered by Year even though it's not a column in the View.
+
+| ID | Title |
+|-|-|
+| 1 | The River Wild |
+| 3 | Daylight |
+| 4 | Jason Bourne |
+| 2 | Don't Look Up |
+
+## Aggregations
+These are functions applied to the view as a whole (rather than to individual columns). 
+
+Using the example table above, imagine a view created from the query `SELECT Genre FROM Movies GROUP BY GENRE;`
+
+| Genre |
+|-|
+| Thriller |
+| Comedy |
+| Action |
+
+## Row Limit
+This is the number of rows the View is limited as per its Query.
+
+## Row Offset
+This is the number of the first row of the result of the Query that the View's rows start at.
+
+
+
+
+
+
+

product/specs/2022-01-views/04-modeling-view-columns.md

@@ -0,0 +1,43 @@
+---
+title: Views: Modeling View Columns
+description: 
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+Here's how I think we should model view columns in our API and UI. Each heading represents an attribute of a View Column.
+
+### Data Type
+- **Definition**: This is the final data type of the content of the column after any computations etc. are applied.
+- **Allowed values**: same as Table data types.
+- **Required**. Data type should always be set, at the very least, we can treat unknown data types as text.
+
+### Sources
+ - **Definition**: This is the set of source columns that are used to generate the data in the current View column.
+- **Allowed values**: references to other Table or View columns, including other columns in the same View.
+- **Optional**: This could be empty for purely calculated columns (e.g. using the Postgres `random()` function and putting the output in a column)
+
+Using Element's UI as an example (Matrix channel names stand in for data sources here), here's how Sources might be represented:
+
+![Sources image](/assets/product/specs/2022-01-views/04-modeling-view-columns/Screen Shot 2022-01-20 at 4.21.05 PM.png)
+
+### Formula
+- **Definition**: This is the formula used to generate data in for this column.
+- **Allowed values**: SQL function + operators
+- **Optional**: Columns that are direct copies of other columns from tables or views won't have a formula.
+
+We should allow users to either use a pre-set set of formulas or (in the future) enter a custom formula using whatever functions are installed on their Postgres database.
+
+Using Element's UI as an example (Matrix channel names stand in for data sources here), here's how a Formula might be represented. Note that Sources are used within the Formula.
+
+![Sources image](/assets/product/specs/2022-01-views/04-modeling-view-columns/Screen Shot 2022-01-20 at 4.23.21 PM.png)
+
+### Link
+- **Definition**: This notes whether a column is a join column. This is a column used to match the same values across multiple tables to create the View. These columns have multiple Sources but no Formula.
+- **Allowed values**: True or False.
+- **Required**: This must be set for all columns.
+
+In [this example View](https://www.w3resource.com/sql/creating-views/create-view-with-join.php), the `agent_code` and `cust_code` columns are Links.

product/specs/2022-01-views/05-ui-requirements-for-views.md

@@ -0,0 +1,84 @@
+---
+title: Product Specs
+description: Views: UI Requirements for Views
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+The first version of Views should support the following features.
+
+I've marked the features we should support as "for alpha release", but I've also listed a few potential future features so that we can think about them while designing code architecture and UI/UX.
+
+# View Setup
+The View Setup functionality should be invoked when creating Views or adding or removing columns to an existing View. What we are doing here is either creating or editing the underlying [Query](/product/specs/2022-01-views/03-modeling-view-query) that defines a View.
+
+View Setup should only be enabled for an existing View if the Query is editable.
+
+View Setup should allow users to:
+- Add or remove [Columns](/product/specs/2022-01-views/04-modeling-view-columns) to the View
+	- If adding, the user should be able to set their Source and Formula
+	- When the column added involves a JOIN, the user should be able to change the type of JOIN used.
+- Add or remove [Query attributes](/product/specs/2022-01-views/03-modeling-view-query)
+
+Here are some *very* rough wireframes to illustrate the idea.
+
+![View Setup image](/assets/product/specs/2022-01-views/05-ui-requirements-for-views/Query Setup.png)
+![View Add Column image](/assets/product/specs/2022-01-views/05-ui-requirements-for-views/View Setup_ Add Column.png)
+
+# Interacting with View Data
+This covers functionality for Views that have already been created.
+
+## Query
+- **For alpha release**: 
+	- Users should be able to see the query associated with a View after it has been loaded.
+	- If the Query is editable, the user could see the broken down attributes.
+- **Potential future features**:
+	- SQL editor for queries
+	- View creation using SQL.
+
+## Columns
+- **For alpha release**: Users should be able to see all columns associated with a view. Each column should show all the [Column attributes](/product/specs/2022-01-views/04-modeling-view-columns), which are non-editable.
+- **Potential future features**:
+	- Show virtual columns involved in view creation (via CTEs, subqueries, etc.)
+	- Allow editing formula through UI
+	- Allow using SQL to edit formula
+
+*Very* rough wireframe to illustrate the idea.
+![View Add Column image](/assets/product/specs/2022-01-views/05-ui-requirements-for-views/View Column Menu.png)
+## Rows
+- **For alpha release**: 
+	- Users should be able to see the rows associated with a view.
+	- If a cell is a direct representation of a record, users should be able to edit that record via that cell. The entire record should be edited through a form, not just the single item.
+		- "Direct representation" means that the record has only one data source, no formula, and is not a Link.
+	- If a column is a direct representation of a record, users should be able to add a new record of the same type via the View UI.
+- **Potential future features**:
+	- "Smarter" editing of Views
+
+## Filters
+- **For alpha release**:
+	- Users should be able to see what filters are applied to their View.
+	- Users should be able to apply new filters to the View.
+	- Users should be able to save filters applied to the View in the UI to the View Query.
+		- Under the hood, this will create a new View with an updated query and delete the old View. In the UI, this will appear as if the View has been updated.
+- **Potential future features**:
+	- Improvements to what columns can be used for filters to support more complex use cases.
+
+## Sorting
+- **For alpha release**:
+	- Users should be able to see what sorts are applied to their View.
+	- Users should be able to apply new sorts to the View.
+	- Users should be able to save sorts applied to the View in the UI to the View Query..
+		- Under the hood, this will create a new View with an updated query and delete the old View. In the UI, this will appear as if the View has been updated.
+- **Potential future features**:
+	- Improvements to what columns can be used for sorts to support more complex use cases.
+
+## Groups
+- **For alpha release**:
+	- Views should support the same groupings as tables based on column data types.
+	- Groups cannot be saved to the database.
+- **Potential future features**:
+	- The ability to save groups as part of a view. This will only work within the Mathesar UI, not at the DB level.
+	- Additional grouping features based on data sources or data formula.

product/specs/2022-01-views/06-mapping-db-queries-to-views.md

@@ -0,0 +1,62 @@
+---
+title: Views: Mapping DB Queries to Views
+description:
+published: true
+date: 2022-01-13T19:49:54Z
+tags: 
+editor: markdown
+dateCreated: 2022-01-13T19:49:54Z
+---
+
+This page goes through the [PostgreSQL documentation about queries](https://www.postgresql.org/docs/14/queries.html) and maps various concepts listed there to our data model. We'll follow the structure of the PostgreSQL docs.
+
+## Table Expressions
+See ["7.2 Table Expressions" on the PostgreSQL docs](https://www.postgresql.org/docs/14/queries-table-expressions.html)
+
+| Clauses | Mapped To | Notes|
+|-|-|-|
+| `FROM` | "Sources" of columns | |
+| `WHERE` | "Filters" | |
+| `GROUP BY` & `HAVING` | "Aggregations" | |
+| `GROUPING SETS`, `CUBE`, & `ROLLUP` | "Aggregations" | |
+| Window function processing | "Formula" of columns | |
+| Join columns | "Link" of columns | |
+
+## Select Lists
+See ["7.3 Select Lists" on the PostgreSQL docs](https://www.postgresql.org/docs/14/queries-select-lists.html)
+
+| Clauses | Mapped To | Notes|
+|-|-|-|
+| Select-List Items | Related to "Sources" of columns | |
+| Column Labels | Used to determine column name in Views | |
+|  `DISTINCT` | "Aggregations" | |
+
+## Combining Queries
+
+See ["7.4. Combining Queries (`UNION`, `INTERSECT`, `EXCEPT`)" on the PostgreSQL docs](https://www.postgresql.org/docs/14/queries-union.html)
+
+There is no direct mapping of query combinations to the Views UI in Mathesar, since they are internal to the query.
+
+They will only be visible when the user looks at the raw SQL query.
+
+## Sorting Rows
+See ["7.5. Sorting Rows (`ORDER BY`)" on the PostgreSQL docs](https://www.postgresql.org/docs/14/queries-order.html)
+
+This maps to View "Sorting".
+
+## LIMIT and OFFSET
+See ["7.6. `LIMIT` and `OFFSET`" on the PostgreSQL docs"](https://www.postgresql.org/docs/14/queries-limit.html)
+
+These map to View "Row Limit" and "Row Offset".
+
+## VALUES Lists
+See: ["7.7 `VALUES` Lists" on the PostgreSQL docs"](https://www.postgresql.org/docs/14/queries-values.html)
+
+These map to View column "Sources". They will show up as a computed source.
+
+## WITH Queries
+See ["7.8. `WITH` Queries (Common Table Expressions)" on the PostgreSQL docs](https://www.postgresql.org/docs/14/queries-with.html)
+
+There is no direct mapping of CTEs to the Views UI in Mathesar, since they are internal to the query.
+
+They will only be visible when the user looks at the raw SQL query.