Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor formula functions and values #532

Merged
merged 21 commits into from
Jun 13, 2023
Merged

Conversation

HactarCE
Copy link
Collaborator

@HactarCE HactarCE commented May 23, 2023

This refactor will enable better Excel compatibility in formula evaluation:

Arrays

  • Independent horizontal & vertical expansion ({1,2,3} + {10;20;30} returns a 3x3 array)
  • SUM(), PRODUCT(), etc. error on invalid single values (SUM(1, "text") returns an error)
  • SUM(), PRODUCT(), etc. ignore invalid values in arrays (SUM(1, {"text"}) returns 1)
  • Error values are propagated, even in arrays (SUM(1, {1/0}) returns an error)
  • Cell references are 1x1 arrays (if A1="text", SUM(1, A1) returns 1)
  • 1x1 arrays returned from operations are converted to single values (if A1 and A2="text", SUM(1, A1&A2) returns an error)

NaN and Inf

  • Division by zero produces a nice pretty error message
  • NaN and Inf in calculations are converted to proper errors

Nulls (possible future behavior)

This is untested for now since we need a more solid type system to make it useful

  • Empty cells contain null
  • null is coerced to zero when used numerically (if A1 is null, then A1+2 returns 2)
  • Empty strings do not coerce to number (if A1 contains a formula producing an empty string, then A1+2 returns an error)
  • null is coerced to an empty string when used as text (if A1 is null, then A1+"text" returns "text")

@HactarCE HactarCE self-assigned this May 23, 2023
@vercel
Copy link

vercel bot commented May 23, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
quadratic ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 13, 2023 3:17pm

@HactarCE HactarCE marked this pull request as ready for review June 5, 2023 18:55
@HactarCE HactarCE changed the title Refactor formulas Refactor formula functions and values Jun 5, 2023
@HactarCE
Copy link
Collaborator Author

HactarCE commented Jun 6, 2023

This also fixes #518

Copy link
Collaborator

@davidkircos davidkircos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@HactarCE HactarCE merged commit 4aa1cef into main Jun 13, 2023
7 checks passed
davidkircos added a commit that referenced this pull request Aug 17, 2023
* fixed local files not properly saving file list

* minor tweaks

* fix black bar in quadrants

* initial work

* copy as png

* lint

* improve menu

* make menus prettier; fix bugs with tranform

* prettier

* fix test

* better vertical centering of header numbers

* make header selection color more pronounced

* Catch TypeError on Decimal Conversion

* increase heading font size

* change Console labels

* update Documentation tab

* Continue to refine DOCUMENTATION tab

* refine docs

* remove support for special variable `result` update examples

* fix unit test

* tweaks

* more tweaks

* pr feedback tweaks

* accept statements as last line for output

* quiet_trailing_semicolon=False

* formatting bar style tweaks

* copy as png icon and command palette

* fix expenses file

* refactor

* tweaks

* anchor context menu in the right location, close moreMenu when FloatingContextMenu closes

* correct

* lint

* crop copy as png perfectly

* more tweaks

* more tweaks/updates

* update examples to be based off main

* remove extra app.save()

* update to support multiple schemas

* hide cell type outlines in png

* prettier

* fixed viewport decelerate behavior when pointerup on negative heading resize

* fix copy as png for large texture sizes

* added snackbar

* prettier

* show loading in CodeEditor

* fix unmatched (

* fix: update context menu icons

* Update CodeEditor.tsx

* properly center spinner

* ignore caps for cell type filter

* initial run at converting typescript types to zod types

* add zod to deps

* rename GridFileSchema type to GridFile

* pull version from the schema

* Update useLocalFiles.ts

* more tweaks

* further tweaks

* Update useLocalFiles.ts

* Update CodeEditor.tsx

* Update CodeEditor.tsx

* Update index.js

* refactor: app structure at load time

* feat: add support for downloading any file in memory

* revert flags change

* refactor: sort type keys optionally, then alphabetically

* Delete FileLoadingComponent.tsx

* rename generic "things"

* tweak download icons

* fixes copy paste encoding bug

* remove unused code

* remove unused prop

* Update FileMenu.tsx

* Update useLocalFiles.ts

* more tweaks

* Update useLocalFiles.ts

* don't throw when you can't find the grid

* change error to warning

* tweaks

* Update useLocalFiles.ts

* Update app.ts

* update typescript

* fix bug with negative heading resizing moving the viewport

* Fix redo icon in Quadratic menu

* Update useLocalFiles.ts

* Update QuadraticMenu.tsx

* fix logo on mobile

* fix firefox bg problem

* refine CMD + O functionality

* Update FileMenu.tsx

* more tweaks

* rename variables

* Update useLocalFiles.ts

* rename pixi to pixi-assets

* Update useLocalFiles.ts

* fixes a bug deleting the wrong cell when deleting a single cell

* bugfix

* initial commit

* Update useLocalFiles.ts

* Update GridFile.ts

* log contents

* update with tests

* Language gpt4 api (#386)

* basic AI support

* refine UI

* refine prompt

* update prompt

* cleanup code

* improve prompt

* Show API errors in console

* show loading in CodeEditor

* add express api

* configure for heroku

* point heroku to subdirectory quadratic-api

* point to correct server

* cleanup

* proxy AI requests via quadratic-api

* update error msg

* add API_URL env var

* make audience optional

* audience remove in api, require audience in react

* set CORS to *

* sentry only on prod

* properly set AUTH0_AUDIENCE

* pass valid AUDIENCE value

* use audience on API

* add ai tab

* style tweaks

* UI improvements

* streaming AI response

* submit on enter, clear prompt

* add some placeholder UI

* hide AI Assistant unless writing python code

* add rate limiting by user

* improvements

* fix margin

* small features

* cleanup frontend, prep api for production

* add docs, hide AI if no auth

* fix import error

* add start log

* fix unit tests

* Attach user to Sentry for error reporting

* update UI, log requests

* remove console.log

* log server errors

* jim's feedback

* update Experimental chip

---------

Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* refactor: file schemas (#403)

* back to files shortcut (#414)

* strip newlines before rendering cell contents (#417)

* Add Amplitude Analytics (#418)

* Add Amplitude Analytics

* update comment

* Update README.md (#420)

* Add MixPanel Analytics (#423)

* add MixPanel analytics

* reduce events fired on load

* recognize 'none' env var as undefined

* fix: redo command for windows (#391)

* Mixpanel Dev Fix (#427)

* feat: add read-only dialog for mobile (#428)

* File Backup on Quadratic Cloud (#425)

* File Backup on Quadratic API

* add prisma command to Procfile

* when backing up file record version and created / modified dates

* add comment

* send JSON response

* lower file size limit

* increase requests allowed to 30 per minute

* feat: add mixpanel people analytics (#431)

* Trigger GitHub Actions on PRs from external contributors (#436)

* Strip currency prefix to mitigate #410 (#411)

Co-authored-by: David Kircos <davidkircos@gmail.com>

* Fix typo in default spreadsheets (#435)

Co-authored-by: David Kircos <davidkircos@gmail.com>

* selection auto complete (#379)

* selection auto complete (WIP)

* fix selecting in open space

* delete cells for autocomplete

* text series and number series expand

* properly finds text series if selection wraps

* copy other cell types besides text

* Update colors.ts (#430)

* tweak drag UI interaction

* fixed bug with expanding empty cells

* fix issues with showing delete cells

* fixing a few more bugs

* fix another bug related to horizonta/vertical selection

* improve horizontal/vertical shrinking

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* feat: preserve `file` query param through auth0 process (#412)

* bug: Fix paste in safari (#438)

* fix: paste behavior in safari

* lint

---------

Co-authored-by: himself65 <himself65@outlook.com>

* fix: indicate "delete" when shrinking horizontally (#439)

* fix: message for failure to load local file (and global snackbar) (#433)

* fix: use `.key` rather than `.code` (#434)

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* feat: add support for `Delete` key (#453)

* Add sentry to api (#459)

* add sentry to API

* better capture errors instead of failing

* add @sentry/tracing to package mgr

* Add sentry to api (#460)

* add sentry to API

* better capture errors instead of failing

* add @sentry/tracing to package mgr

* fix file access error

* fix: allow wider array of characters for initial cell input (#451)

* chore: update browserslist (#461)

* Dont call captureException with a INFO level message (#464)

* chore: support hosting via vercel (#467)

* fix: gridSettings (#470)

* feat: AI ui/ux enhancements (#466)

* fix: support spaces when importing file from url

* fix: close editor when creating new file (#465)

* fix: changing settings properly rerenders cells (#475)

* fix: bug with upgrade file in validateGridFile (#476)

* Implement range operator `a..b` (#444)

* Improve formula parse error messages
* Implement `**` formula operator
* Bump Rust version

`<<`, `>>`, and `**` don't exist in Excel so they aren't implemented.

Well `**` sort of exists, but it's used for scientific notation which is
cursed as heck and I don't want to change float parsing right now.

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* Fix async grid access during formulas + refactors (#360)

* Reorder formula functions to match official docs
* Refactor formulas
* Add `col!` and `pos!` macros
* Add `parse_formula()` wasm function
* Expose `column_name` and `column_from_name` to JS
* Bump quadratic-core version
* Replace `CELL` with Excel-compatible `INDIRECT`

This fixes a number of issues:
- Grid access no longer blocks, so `INDIRECT` function should actually work
- Documentation is now included alongside function definitions, so we can auto-generate formula function documentation soon

* feat: save cursor position with undo/redo (#455)

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* feat: implement LSP functions in Rust (#477)

* Refactor formulas
    * Reorder formula functions to match official docs
    * Add some missing formula documentation
    * Documentation is now included alongside function definitions, so we can auto-generate formula function documentation soon
    * Add examples for all formula functions
* Add `parse_formula()` wasm function
* Expose `column_name` and `column_from_name` to JS
* Implement LSP autocomplete via Rust
* Replace `CELL` with Excel-compatible `INDIRECT`
    * Make grid access async so it should actually work now
* Autogenerate Notion docs for formulas

quadratic-core internal changes:
* Add macro for JS value manipulation
* Add `col!` and `pos!` macros

* refactor: move some formula tests to specific files (#498)

* Fix formulas not automatically updating (#499)

* Fix formulas not automatically updating

* feat: new example files (#503)

* fix: handle escape while filename renaming (#495)

Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* feat: selection auto complete multiple cells (#452)

* Add regression test for #499 (#501)

* Add regression test for #499

* Implement `ABS`, `SQRT`, `PI`, `TAU`, and trigonometric functions (#497)

* Implement trigonometric functions
* Add `ABS`, `PI`, and `TAU`
* Add `DEGREES` and `RADIANS` functions
* Add `SQRT`

* chore: remove debug menu (#508)

* Ignore empty values in variadic formula functions (`SUM`, `PRODUCT`, `COUNT`, and `AVERAGE`) (#507)

* Remove `**`, which shouldn't have been in there anyway
* Fix `SUM`, `PRODUCT`, and `COUNT` semantics
* Test formula math operators with empty string

* Alignment (#318)

* fix up cellLabels

* automatic number alignment

* overflow right

* alignment to the left

* cleaning up

* alignment

* fix jumping text

* fix formatting

* additional fixes

* apply prettier

* cleans up flags

* make FloatingContextMenu a bit smaller

* remove dependency loading (caused bugs)

* cleaning up floating menu

* prettier

* fix bugs

* fix bugs with debug elements when in presentation view

* apply prettier

* fix isNaN function for strings

* standardize usage of isStringANumber for data formatting

* fix alignment calculated based on original value, not formatted value

* fix clipping; remove axis from heading resize

* prettier

* reset debugFlags

* remove debug setting from .env

* fixed cellLabels

* lint

* update examples formatting

* fix python.grid

* update schema

* remove render_dependency in v1_2 file

* cleaning up after removing GridRenderDependency

* update readme to trigger build

* use examples from main

* Update README.md

* fix capitalization

* remove right alignment for numbers

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* Fixes bug with v1_2 format (#521)

* Fixes bug with v1_2

* place updateFn in correct place

* fix tests; add v1_1 -> v1_2 test

* fix: delete empty cells not in undo/redo stack (#494)

* fix: autocomplete formatting (#524)

* Fixes bug with v1_2

* place updateFn in correct place

* fix tests; add v1_1 -> v1_2 test

* added formatting

---------

Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* fix: typo in FileMenuTabs.tsx (#500)

Changed "file format that be saved" to "file format the is saved"

* feat: confirm unsaved changes (#525)

Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* fix clipping calculation to the right (#523)

Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* docs: improve README and CONTRIBUTING (#527)

* better resolution for quadrants for windows (#542)

* Refactor overlap detection (#543)

* redoing clip calculation

* clipping improvements

* initial commit (#538)

Co-authored-by: David Kircos <davidkircos@gmail.com>

* feat: autocomplete borders (#548)

* Double click to size column to fit (#546)

* function to get cells content width

* implement column to fit

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>

* feat: add Open AI example file (#531)

* Update app.ts

* Update app.ts

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>

* Improve performance update cells (#529)

* use sets instead of array

* removed duplicated line

* create new helper function and use it

* Update getKey.ts

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>

* feat: add a feedback widget (#450)

* initial commit

* start

* tweaks

* Update feedback.ts

* Update debugFlags.ts

* move logic into api layer

* add v0 prefix for feedback

* remove email

* add minrows

* don't show feedback on mobile

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>

* feat: move command palette out of header into main menu (#535)

* fix: 489

* updates

* fix: support null in array_output file schema (#491)

* Horizontal scroll (#533)

* implement horizontal scroll

* use key consts

* disable on mac

* use utils isMac

---------

Co-authored-by: David Kircos <davidkircos@gmail.com>

* feat: replace input with div[contenteditable] (#454)

* fix: properly resize negative rows  (#553)

Fixes #550

* feat: double click to reset row (#552)

* Refactor formula functions and values (#532)

* Remove `**`, which shouldn't have been in there anyway
* Add more tests for criteria strings

I also caught a bug: escaping using `~` wasn't working for `?` and `*`.

* Add a bazillion tests
* Fix `SUM`, `PRODUCT`, and `COUNT` semantics
* Test formula math operators with empty string
* Refactor formula function definitions
* Improve div-by-zero error message
* Test 1x1 array cell reference behavior

* fix: change schema to support wider variety of array_output (#561)

* Add `AVERAGEIF`, `COUNTIF`, `SUMIF`, and `COUNTBLANK` (#492)

* refactor: improve selection and misc. performance (#545)

* fix: clip when cell has error (#554)

* chore: remove extra focusGrid call (#562)

* fix: sentry errors (#563)

* feat: post user feedback to slack (#566)

* feat: update BottomBar.tsx (#573)

* Allow omitting arguments in formula functions (#555)

* Fix `ATAN2(0, 0)` semantics (#571)

* Fix `ATAN2(0, 0)` semantics
* Document `ATAN(0, 0) = 0`

* Add boolean parsing in formulas (#572)

* Add boolean parsing in formulas
* Test bool parsing
* Omit parens in autocomplete for `TRUE` and `FALSE`

* fixes app.settings.setInteractionState (#577)

* chore: upgrade API to node 20 and typescript 5 (#574)

* Add `HLOOKUP`, `VLOOKUP`, and `XLOOKUP` (#569)

* fix: allow any for array_output in older versions of the schema (#580)

* fix: allow any for array_output

* Update validateGridFile.ts

* Update validateGridFile.ts

* Update validateGridFile.test.ts

* fix: copying of large data (#588)

* feat: better zero states around code (#541)

* tweaks (#596)

Co-authored-by: Hactar <6060305+HactarCE@users.noreply.github.com>

* feat: enhance AI assistant ui/x (#583)

* feat: mixpanel proxy (#586)

* fixed typo and bump index (#560)

Co-authored-by: Jim Nielsen <jimniels@gmail.com>
Co-authored-by: David Figatner <david@yopeyopey.com>

* fixed selection (#557)

before this change, selecting a cell past (-999,0) in the -x direction resulted in a mis-selection by 1.

This manually fixes this issue.

I discovered a similar issue with -y values past (0,-999) that multiplies the selection by 5.

Co-authored-by: David Figatner <david@yopeyopey.com>

* feat: show active cell selection formula results in bottom bar (#594)

* Update BottomBar.tsx

* Update BottomBar.tsx

* Update BottomBar.tsx

* Update BottomBar.tsx

* add console

* Update BottomBar.tsx

* Fix COUNT semantics add COUNTA

* Bump Rust version

* Use COUNTA formula in bottom bar

* move ActiveSelectionStats to a new component

* further clean up code

* Do not compute stats for >100 cells

* Increase selection stats limit to 250

---------

Co-authored-by: HactarCE <6060305+HactarCE@users.noreply.github.com>
Co-authored-by: David Kircos <davidkircos@gmail.com>

* increase timeout for font loading (#607)

* python web-worker port (#587)

* chore: add setting so VSCode automatically sorts imports #612

* Highlight cells referenced in code (#559)

* add cell reference multicursor

* create hook to extract tokens from model

* add cell highlight colors

* add helpers to parse formulaNotation

* add hook and state to check if ref valid

* add cell highlights to editor interaction atom

* add method to draw highlights

* Add range to cell matches

* remove highlighted cell from editorInteractionState

* create new atom for highlighted cells

* create hook for editor selection change

* add new hook to codeEditor

* add new state to pixiappsettings

* add cell fill on selection

* add update for editorHighlightedCells

* fix expression to exclude $

* fix larger cell values

* design tweaks

* fix prettier issues

* export function

* refactor to parse_formula

* remove parser and add return type

* use new modified function

* remove unused regex

* Update colors.ts

* add python cell parser

* add selectedCellType to hook

* add python parser and fix cell highlight on change range

* change type

* Reverts python parser commits
5af393b
daec383
ae056a9

* fix selected cell change fill

---------

Co-authored-by: Jim Nielsen <jimniels@gmail.com>
Co-authored-by: David Kircos <davidkircos@gmail.com>

* fixes bug #611 (#621)

---------

Co-authored-by: David Figatner <david@yopeyopey.com>
Co-authored-by: Jim Nielsen <jimniels@gmail.com>
Co-authored-by: HactarCE <6060305+HactarCE@users.noreply.github.com>
Co-authored-by: Peter Mills <waveminded@users.noreply.github.com>
Co-authored-by: Bart Broere <mail@bartbroere.eu>
Co-authored-by: himself65 <himself65@outlook.com>
Co-authored-by: Nicholas <63423605+Nerptastic@users.noreply.github.com>
Co-authored-by: Juan Manuel Bercoff <jmbercoff@gmail.com>
Co-authored-by: ajboots <98557530+ajboots@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants