Audit Logs for Postgres connections opened through a data link

[radeusgd]

Pull Request Description

• Closes Created audit logged JDBC connection for Postgres based Datalinks #9599
• Implemented API for sending audit logs to the cloud on a background thread.
• If the Postgres connection is opened through a datalink, its internal JDBC connection is replaced by a wrapper that reports executed queries to the audit log.
• Also introduces EnsoMeta - a helper Java class that can be used in our helper libraries to access Enso types.
  • I have replaced the common pattern scattered throughout the codebase with calls to this 'library' to avoid repetitive code.
• Refactored Table.display to share code between in-memory and DB - it was needed as the function stopped working for DB_Table after adding making the Table constructor private.
• Clearer error when reading a SQLite database from a remote file (tells the user to download it first).
• Follow up - correlate asset id of the data link:
  Correlate Database audit log entries with the asset id of the data link used for the connection, if available #9869
• Follow up - include project name (once bug is fixed):
  Include project name in Database audit logs #9875
• Some problems/improvements of the audit log:
  • The audit log system is not yet ready for high throughput of logs
    Audit Logs should be able to handle pressure #9870
  • The logs may be lost if System.exit is used
    System.exit does not give a chance for background threads to complete, affecting audit logs #9871

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
• All code follows the
  Scala,
  Java,
  TypeScript,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• Unit tests have been written where possible.

[radeusgd]

Currently metadata is not audited. That means that the act of listing tables existing in the database is not audited.

Do you think we should change that? It was unclear to me if it's something that we want to log or not. It is easy to add if we decided that we do want it.

[jdunkerley]

Agree don't think we need to audit Metadata queries.

[radeusgd]

I was curious what happens if I open an SQLite database file stored in the Enso Cloud (or S3).

It was failing with Method read_stream of type SQLite_Format could not be found..

So I added the missing method, changing the failure to a clearer: Illegal_Argument.Error 'Cannot connect to a SQLite database backed by a stream. Save it to a local file first.'

[jdunkerley]

Looks great - a few little questions.

[jdunkerley]

Agree don't think we need to audit Metadata queries.

[jdunkerley]

How come this needs to be in Standard.Base?

[radeusgd]

The problem is that Enso_Cloud.Internal.Utils used here are private.

We'd need to start running Base_Tests with --disable-private-check. But tbh I'm worried about doing that as I already noted somewhere. If we run our whole test suite with --disable-private-check then we are testing with different settings than used in production. We already ran into problems where private keyword introduced regressions - the display method that I fix in this PR got broken because of it. Here the regression went unnoticed because our tests did not cover the display method. But if we then --disable-private-check for our test suite, we can have such undetected regressions in other places, only to be seen when used in production environments that lack the --disable-private-check parameter.

On the other hand I remember we wanted to restructure our tests in such a way that they'd be able to access private methods of the tested package by default. Given that Enso is an inherently dynamic language, the problems described in the paragraph above convince me this may not be the safest thing to do.

[radeusgd]

Still, I agree that these utils probably should not be part of the library.

I can move them to the tests and change so that Base_Tests are run with --disable-private-check for now. But I think for longer term some better solution is needed.

My proposal (that's just one way we could do that) would be to:

• change --disable-private-check to --allow-disabling-private-check and changing the methods of overcoming private acccess to be more localized:
• for the static part - i.e. imports - introduce a private import A.B.C syntax that allows to import private modules,
• for the dynamic part, introduce Context.Private_Access.with_enabled that allows to enable private access only within a delimited scope.

The two new constructs would only be allowed if --allow-disabling-private-check flag was passed to the interpreter. If no such flag was passed, the private import would need to result in a compiler error, and the Context.Private_Access.with_enabled would panic. Thus in normal usage of Enso, private access will not be allowed, and within our tests (that would have this flag turned on), we can enable the access but only if explicitly opted-in, delimited to the minimal scopes where this is really needed (e.g. only to implement such Test_Utils helper). Thus all other tests that do not rely on exposing privacy would be unaffected and still test the behaviour as it is working in production configurations.

[radeusgd]

Report from OpenSearch logs, sorted by the sequence number. I highlighted a block corresponding to a table update rows transaction.