Proper implementation of Value Types in Table

[radeusgd]

Pull Request Description

This is the first part of the #5158 umbrella task. It closes #5158, follow-up tasks are listed as a comment in the issue.

• Updates all prototype methods dealing with Value_Type with a proper implementation.
• Adds a more precise mapping from in-memory storage to Value_Type.
• Adds a dialect-dependent mapping between SQL_Type and Value_Type.
• Removes obsolete methods and constants on SQL_Type that were not portable.
• Ensures that in the Database backend, operation results are computed based on what the Database is meaning to return (by asking the Database about expected types of each operation).
  • But also ensures that the result types are sane.
    • While SQLite does not officially support a BOOLEAN affinity, we add a set of type overrides to our operations to ensure that Boolean operations will return Boolean values and will not be changed to integers as SQLite would suggest.
    • Some methods in SQLite fallback to a NUMERIC affinity unnecessarily, so stuff like max(text, text) will keep the text type instead of falling back to numeric as SQLite would suggest.
• Adds ability to use custom fetch / builder logic for various types, so that we can support vendor specific types (for example, Postgres dates).

Important Notes

• There are some TODOs left in the code. I'm still aligning follow-up tasks - once done I will try to add references to relevant tasks in them.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
  • Check Table.info in GUI.
  • Check SQL visualization in GUI.
• All code follows the
  Scala,
  Java,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• All code has been tested:
  • Unit tests have been written where possible.
  • If GUI codebase was changed, the GUI was tested when built using ./run ide build.

[radeusgd]

1. Verified that the SQL visualization still works.
2. Added to_js_object override to ensure that Value Type is visualized nicely in Table.info displayed in the IDE.

(Note how the color of the interpolated parameter in the SQL query visualization matches the color of its Enso type (Integer and Text) - no changes here, it's the thing we did almost 2 years ago, but I just like it a lot 😁)

[GregoryTravis]

Just curious, is this dummy parameter necessary to avoid early evaluation of lazy_computation?

[radeusgd]

Tbh I did not test it, but I'm pretty sure that fields are always stored 'eagerly' in an Atom - and the builder will be stored in an Atom a few lines later.

So to store a 'thunk' I need to wrap it in a lambda.

[JaroslavTulach]

Fields will always be stored "eagerly" in the Atom until we implement

• Provide metadata for widgets lazily #5085

Would you benefit from having "lazy atom fields"? Ask and we'll be more than glad to implement it!

[JaroslavTulach]

Frgaal upgrade is certainly OK.

[JaroslavTulach]

I'd prefer implementation of "lazy/cached atom fields" in the engine rather than adding yet another (to Ref type) low level support type into Standard.Base.

[JaroslavTulach]

I believe there should be no Lazy type in the libraries, but the support shall come from the engine:

• Provide metadata for widgets lazily #5085
• we have a prototype that shows how it all can be done
• the API will be simpler - just ~ in front of the atom field
• the access to the field will be transparent - no difference between lazy/cached/eager field
• the performance will be better if this is implemented in the engine
• we'll use less memory, if this is done in the engine as we eliminate delegation once the field is materialized/cached

Can we get a request from libraries to implement "lazy atom fields"?

[radeusgd]

Ok, sounds all good!

I implemented this in pure Enso because I could do it quickly (it took about 1-2hr) and didn't want to wait for it to be implemented, as I wanted to be able to rely on this code straight away.

It also felt like it is simpler to do that way and at least for now we don't need this to be super performant - it's not used on some very hot paths - so I didn't want to take engine's time for something not critical.

But if we can get it replaced with a more efficient engine solution - I will be really happy to have it replaced. The benefits sound cool.

[radeusgd]

However, I'd like to proceed with this PR without waiting for this implementation. So I would just keep the Lazy type for now. But once it is implemented in the engine, I'm more than happy to have it replaced with the "native" solution.

[JaroslavTulach]

Reported as

• Easy way to delay & cache computed atom fields #6134

[JaroslavTulach]

Am I right this is the core part of the PR? At least the file name Table/**/Value_Type.enso matches the title "..Value Types in Table..".

Shouldn't there be an example of how the Value_Type is supposed to be used from an end user perspective?

• I can envision some form of conversions from/to Enso types.
• TruffleObject provides special support for converting of objects between systems/languages/types
• Wrapping the values into Enso Value_Type is unlikely the most effective way
• but I'd need to understand the use-case more...

[radeusgd]

Yep, you are right about the file being important :)

As for examples, I guess more will be available once we implement functions like cast. For now you can see how it's used in Table_Spec or for example Postgres_Spec.

Currently we are not performing conversions to/from Enso types, at least not directly, the concepts are a bit parallel. We could do these, but there was no use-case yet.

As for TruffleObject and mappings - I'm not sure we need this either. Sounds interesting, maybe could be useful if we have a usecase.

Currently, the Value_Type is meant to provide metadata of what kind of values may be stored in a Table Column. It is tailored to work well with both in-memory and Database columns, so the type system is built around how SQL types tend to work.

[jdunkerley]

This looks like a fantastic job - I'm through 86 of the files, and will review the rest tomorrow.

[jdunkerley]

While I agree the todo - think the line should be enabled?

[radeusgd]

Without it, BigDecimal will be just stored as an object and the column will get a Mixed type. Won't that be better until we get proper support?

[radeusgd]

We don't have a dedicated storage type for BigDecimal anyway. Nor for BigInteger which is more problematic since we have a real "bug" here because BigIntegers are natively supported by Enso (an Enso Integer can be a long or BigInteger.

I'm not sure the retyping will work correctly if I just enable it, it would require additional work. Let's keep it as-is for now. If we want support for BigDecimal in Table, let's add a ticket and do it properly.

[radeusgd]

For the record, this is what currently happens if I put a BigInteger into a Table - it's not bad, because since that type is unrecognized, it is just treated as a Mixed column:

Not ideal, because we cannot do arithmetic operations on it directly (since it is not of a numeric type). But at least there's no data corruption.

[jdunkerley]

LGTM - a great piece of work.
A few things to look over and think on but generally awesome.