Expand capabilities of Table.set and better dropdown support,

[jdunkerley]

Pull Request Description

• Adds the ability to use numbers, date/time and Boolean values as constants in set.
• Table.set can take a Column_Operation, allowing for deriving of a new column based on other columns.
• Added Column_Ref type to refer to a column in filter.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
• All code follows the
  Scala,
  Java,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• All code has been tested:
  • Unit tests have been written where possible.
  • If GUI codebase was changed, the GUI was tested when built using ./run ide build.

[radeusgd]

ToDo

• Add Is_In and Not+In
• Add Mod and Power Column_Operation wrapping % and /.
• Add Trim to Column_Operation.
• Add all the missing tests 🙂
• If needs default constants for true and false args
• Switch input to be a Column_Ref or Any - follow the same logic as in set but no support for an expression or a Column_Operation.
• Add Min and Max as binary mode operations.
• Verify widgets work:
  • Is_In / Not_In
  • Min / Max
  • Mod / Power
  • Trim
  • format

Follow up tickets

• Add ticket for equals_ignore_case, is_nan and is_infinite on Filter_Condition.
  Additional Filter_Conditions for is_nan, is_infinite and equals_ignore_case #8045
• Review the places we could use Column_Ref to allow easy reference.
  • e.g. Table.fill_nothing, Table.fill_empty
  • Review places where we can start using Column_Ref #8046
• Inconsistency between in-memory and DB truncate - in-memory allows Numbers and Date_Time, DB Does not allow datetime.
  Implement truncate Date_Time for database backend #8047
• Add format to Table with same kind of API as parse.
  Implement Table.format similar to Table.parse allowing to format columns in bulk #8048
• Avoid Inexact_Type_Coercion when Date_Time False gets converted to Date_Time True - this is just too common and annoying vs useful - e.g. when loading Postgres table whose tzs don't have a timestamp.
  Avoid Inexact_Type_Coercion when downloading a database table #8049
• Consider better naming for Constant columns:
  Constant columns (in expressions and Column_Operations) should have clearer names #8050

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2023-10-12):

Progress: Added missing parts and tests to derive columns. It should be finished by 2023-10-13.

Next Day: Next day I will be working on the same task. Do some cleanup, verify widgets work. Create followup tasks.

[radeusgd]

I added this check that the values do not contain a Column_Ref, because IIRC we have agreed that there is no sense to provide a column selector widget for this case - as it can be confusing. So I guess there is neither sense to resolve the Column_Refs here.

(The confusion is that normally operations work row-by-row but Is_In, when provided a column will look if the values on the left are contained anywhere within the column on the right - so it's a different logic.)

So I guessed that since we do not resolve the Column_Ref, we should not accept it either - otehrwise it would be compared by equality and just yield to unexpected results.

So I added this errror:

Column_Ref is not allowed in Is_In to avoid unexpected behavior. As opposed to other operations, which operate on a row-by-row basis when passed a column, Is_In looks at the whole contents of the passed collection - thus usually expecting a Vector. If you want to filter the elements checking if they are present anywhere in a passed column, you can pass a column by first getting it from the table using the at operator.

IMO it explains OK why by default this is not allowed and how the user can work around it if they know what they are doing (use at).

Please note that in normal flow, the user should never see this error - the widget for Is_In does not suggest selecting column references - it includes blank _ nodes instead waiting for input. The error will only show up if the user manually types in Column_Ref.Name "..." for this argument.

[radeusgd]

I moved Column_Ref to a separate file, because I think it may be used in other contexts as well so it seemed sensible to have it in its own file.

[jdunkerley]

👍

[jdunkerley]

Thanks Radek this looks great.

[jdunkerley]

👍

[enso-bot]

Radosław Waśko reports a new STANDUP for the provided date (2023-10-13):

Progress: Created followup tasks. Cleanup; got the PR merged. Work on optimizing add row number. It should be finished by 2023-10-13.

Next Day: Next day I will be working on the #8055 task. Add another benchmark and see potential reasons for slowness, summarize. Get the Problem handling refactor merged