New set function, parse a column #4097

jdunkerley · 2023-01-30T09:44:31Z

Pull Request Description

New set function design - takes a Column and works with that more easily and supports control of Set_Mode.
New simple parse API on Column.
Separated expression support for filter to new filter_by_expression on Table.
New compute function allowing creation of a column from an expression.
Added case sensitivity argument to Column based on starts_with, ends_with and contains.
Added case sensitivity argument to Filter_Condition for Starts_With, Ends_With, Contains and Not_Contains.
Fixed the issue in JS Table visualisation where JavaScript date was incorrectly set.
Some dynamic dropdown expressions - experimenting with ways to use them.
Fixed issue with .pretty that wasn't escaping \.
Changed default Postgres DB to postgres.
Fixed SQLite support for starts_with, ends_with and contains to be consistent (using GLOB not LIKE).

Checklist

Please include the following checklist in your PR:

The documentation has been updated if necessary.
All code conforms to the
Scala,
Java,
and
Rust
style guides.
All code has been tested:
- Unit tests have been written where possible.
- If GUI codebase was changed: Enso GUI was tested when built using BOTH
  ./run ide build and ./run ide watch.

farmaazon

Approval of table visualization file.

engine/runtime/src/main/java/org/enso/interpreter/runtime/data/text/Text.java

hubertp · 2023-01-31T13:46:21Z

distribution/lib/Standard/Base/0.0.0-dev/src/Data/Filter_Condition.enso

+    ## PRIVATE
+       Gets a widget set up for a Filter_Condition.
+    widget_for_filter_condition =
+        ## values = ["(Filter_Condition.Equal to=_)", "(Filter_Condition.Not_Equal to=_)", "(Filter_Condition.Is_In values=_)", "(Filter_Condition.Not_In values=_)", "Filter_Condition.Is_True", "Filter_Condition.Is_False", "Filter_Condition.Is_Nothing", "Filter_Condition.Not_Nothing", "Filter_Condition.Is_Empty", "Filter_Condition.Not_Empty", "(Filter_Condition.Less than=_)", "(Filter_Condition.Equal_Or_Less than=_)", "(Filter_Condition.Greater than=_)", "(Filter_Condition.Equal_Or_Greater than=_)", "(Filter_Condition.Between lower=_ upper=_)", "(Filter_Condition.Starts_With prefix=_)", "(Filter_Condition.Ends_With suffix=_)", "(Filter_Condition.Contains substring=_)", "(Filter_Condition.Not_Contains substring=_)", "(Filter_Condition.Like pattern=_)", "(Filter_Condition.Not_Like pattern=_)"]


Is this commented out value still needed?

Preferred this form but it caused IDE issues so stuck with the simpler form.
Kept for now in case want it as widgets develop.

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Helpers.enso

CHANGELOG.md

radeusgd · 2023-01-31T11:56:33Z

distribution/lib/Standard/Base/0.0.0-dev/src/Data/Filter_Condition.enso

+    ## PRIVATE
+       Gets a widget set up for a Filter_Condition.
+    widget_for_filter_condition =
+        ## values = ["(Filter_Condition.Equal to=_)", "(Filter_Condition.Not_Equal to=_)", "(Filter_Condition.Is_In values=_)", "(Filter_Condition.Not_In values=_)", "Filter_Condition.Is_True", "Filter_Condition.Is_False", "Filter_Condition.Is_Nothing", "Filter_Condition.Not_Nothing", "Filter_Condition.Is_Empty", "Filter_Condition.Not_Empty", "(Filter_Condition.Less than=_)", "(Filter_Condition.Equal_Or_Less than=_)", "(Filter_Condition.Greater than=_)", "(Filter_Condition.Equal_Or_Greater than=_)", "(Filter_Condition.Between lower=_ upper=_)", "(Filter_Condition.Starts_With prefix=_)", "(Filter_Condition.Ends_With suffix=_)", "(Filter_Condition.Contains substring=_)", "(Filter_Condition.Not_Contains substring=_)", "(Filter_Condition.Like pattern=_)", "(Filter_Condition.Not_Like pattern=_)"]


What is the purpose of the commented out line? If we want to include it, I'd suggest explaining what this alternative code means.

Kept for now as an alternative experiment. I don't expect either of these lines to be here for much longer once we get to proper structure being passed.

distribution/lib/Standard/Base/0.0.0-dev/src/Metadata.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Column.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Postgres_Options.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Column.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso

radeusgd · 2023-01-31T12:48:29Z

distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Postgres/Postgres_Dialect.enso

@@ -242,7 +242,7 @@ agg_count_distinct_include_null args =

 ## PRIVATE


I was surprised to learn that while Postgres has starts_with, there's no ends_with 😮

Yep, stoopid.

radeusgd · 2023-01-31T12:49:16Z

distribution/lib/Standard/Database/0.0.0-dev/src/Internal/SQLite/SQLite_Dialect.enso

-    text = [starts_with, contains, ends_with, fold_case, make_case_sensitive]+concat_ops
+    text = [starts_with, contains, ends_with, make_case_sensitive]+concat_ops


Why is it removed?

Using the dialect was causing other issues with making the behaviour between the 3 (SQLite, Postgres, InMemory) act consistently. Not using and relying on the LOWER(UPPER(...)) produced consistent results.

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso

radeusgd

Awesome changes! 🎉

I'm really glad on how the new Table.set is looking, I think it's a good direction. Column.parse and Table.compute are also very cool.

Just some comments in line, I think it would be great to add a few tests.

distribution/lib/Standard/Base/0.0.0-dev/src/Metadata.enso

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso

distribution/lib/Standard/Table/0.0.0-dev/src/Data/Column.enso

distribution/lib/Standard/Table/0.0.0-dev/src/Data/Table.enso

radeusgd · 2023-01-31T16:00:59Z

distribution/lib/Standard/Table/0.0.0-dev/src/Data/Table.enso

+        new_column = Expression.evaluate expression get_column make_constant "Standard.Table.Data.Column" "Column" Column.var_args_functions
+        if new_column.is_error then new_column else
+            warnings = Warning.get_all new_column
+            rename = new_column.rename expression


I see that here we are not 'sanitizing' the column name. Shall we try to make the generated column names consistent? Maybe let's sanitize them on both sides then? Ideally using a shared function, probably from a shared module like Table_Helpers.

I remember we had some task to do at some point to ensure that names of columns generated by operations on columns (like + etc.) should be consistent between the two backends (currently they are not); so ideally these new operations should not introduce new inconsistencies.

distribution/lib/Standard/Table/0.0.0-dev/src/Errors.enso

test/Table_Tests/src/Common_Table_Operations/Core_Spec.enso

radeusgd · 2023-01-31T16:17:29Z

test/Table_Tests/src/Formatting/Parse_Values_Spec.enso

+            t2.at "floats" . to_vector . should_equal c2
+            t2.at "bools" . to_vector . should_equal c3
+
+    Test.group "Column.parse" <|


I think I don't see any tests for column.parse Auto. Shall we add one?

Make postgres default DB be postgres.

Adjust SQLite to use GLOB for starts_with and ends_with to make it case sensitive by default.

DB implementation is still old but API matches.

jdunkerley force-pushed the wip/jd/column-parse branch 2 times, most recently from 6621ad1 to 1ad5f13 Compare January 31, 2023 11:01

jdunkerley marked this pull request as ready for review January 31, 2023 11:01

jdunkerley requested review from radeusgd, 4e6, JaroslavTulach, hubertp, MichaelMauderer, wdanilo and farmaazon as code owners January 31, 2023 11:01

farmaazon approved these changes Jan 31, 2023

View reviewed changes

JaroslavTulach approved these changes Jan 31, 2023

View reviewed changes

engine/runtime/src/main/java/org/enso/interpreter/runtime/data/text/Text.java Show resolved Hide resolved

jdunkerley force-pushed the wip/jd/column-parse branch from 21b91f8 to 22aa2e3 Compare January 31, 2023 14:26

hubertp approved these changes Jan 31, 2023

View reviewed changes

radeusgd reviewed Jan 31, 2023

View reviewed changes

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso Outdated Show resolved Hide resolved

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso Outdated Show resolved Hide resolved

radeusgd reviewed Jan 31, 2023

View reviewed changes

distribution/lib/Standard/Database/0.0.0-dev/src/Data/Table.enso Outdated Show resolved Hide resolved

radeusgd approved these changes Jan 31, 2023

View reviewed changes

jdunkerley force-pushed the wip/jd/column-parse branch from 2cc81a4 to 80ec6c0 Compare January 31, 2023 18:22

jdunkerley added 11 commits January 31, 2023 19:44

Initial version of Column.parse.

a13255e

Tweak for widgets test.

fe29bd9

Looking at starts_with on a column.

b1a58ac

Add some more dropdowns for database stuff.

e1668eb

Make postgres default DB be postgres.

Add case sensitivity controls to Starts_With.

2310fcc

Sorting out Column.parse

23334d6

Tests for Case Insensitive functions.

503a181

Adjust SQLite to use GLOB for starts_with and ends_with to make it case sensitive by default.

Avoid use COLLATE for case insensitive in SQLite

d56a2d0

Update set API to new design.

746e359

DB implementation is still old but API matches.

Table Date rendering fix.

d53a4c6

Fix pretty issue where \ not escaped.

804dcbd

jdunkerley added 15 commits January 31, 2023 19:47

Fix Viz tests and update Database set.

aca6676

Fix rename on column.

d4351c6

Separate expressions from filter into filter_by_expression.

246d781

Working on tests for filter by expression.

78c569b

Fix failure tests for filter_by_expression.

04e4a49

Add Filter_Condition Case Sensitivity controls and tests.

5ccee12

Add Filter_Condition Case Sensitivity tests for List and Vector.

0b62b68

Add parse tests.

1049d94

Add set tests.

3eb9a7f

Add set tests.

67ac8bf

Changelog.

87da16f

Prettier and failing tests.

43a7e5e

Fix weird db test...

4fc9036

PR Comments.

2d6c524

PR Comments.

2baad65

jdunkerley force-pushed the wip/jd/column-parse branch from 80ec6c0 to 2baad65 Compare January 31, 2023 19:50

jdunkerley added the CI: Ready to merge This PR is eligible for automatic merge label Jan 31, 2023

mergify bot merged commit 0790ce4 into develop Jan 31, 2023

mergify bot deleted the wip/jd/column-parse branch January 31, 2023 20:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New set function, parse a column #4097

New set function, parse a column #4097

jdunkerley commented Jan 30, 2023 •

edited

Loading

farmaazon left a comment

hubertp Jan 31, 2023

jdunkerley Jan 31, 2023

radeusgd Jan 31, 2023

jdunkerley Jan 31, 2023

radeusgd Jan 31, 2023

jdunkerley Jan 31, 2023

radeusgd Jan 31, 2023

jdunkerley Jan 31, 2023 •

edited

Loading

radeusgd left a comment

radeusgd Jan 31, 2023

radeusgd Jan 31, 2023

		@@ -242,7 +242,7 @@ agg_count_distinct_include_null args =

		## PRIVATE

		text = [starts_with, contains, ends_with, fold_case, make_case_sensitive]+concat_ops
		text = [starts_with, contains, ends_with, make_case_sensitive]+concat_ops

New set function, parse a column #4097

New set function, parse a column #4097

Conversation

jdunkerley commented Jan 30, 2023 • edited Loading

Pull Request Description

Checklist

farmaazon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdunkerley Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

radeusgd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdunkerley commented Jan 30, 2023 •

edited

Loading

jdunkerley Jan 31, 2023 •

edited

Loading