Extension Proposal: [Change tracking via Carbonite]

[terrisgit]

** What is the purpose of the extension? **

Track changes to resources that are stored in Postgres tables using the bitcrowd/carbonite project. (Only Postgres 13 and higher are supported.) See https://hexdocs.pm/carbonite/Carbonite.html for the list of benefits and how it works.

** How would it extend the DSL? **

I am open to suggestions. In my case every resource would be tracked but in the general case only certain resources should be tracked.
"track changes with carbonite"

** What it would do? **

1. Would these be manual tasks?

A) Add Carbonite dependencies. See https://hexdocs.pm/carbonite/Carbonite.html#module-hex-dependency

B) Create a migration to add the carbonite schema if it doesn't already exist. Perhaps the dsl should accept an optional schema name. See https://hexdocs.pm/carbonite/Carbonite.html#module-creating-the-initial-migration

2. Create a migration to add triggers to the table for cud operations - see https://hexdocs.pm/carbonite/Carbonite.html#module-trigger-configuration

3. Alter all cud operations to create a Carbonite transaction row. See https://hexdocs.pm/carbonite/Carbonite.html#module-inserting-a-transaction

** Interaction with other features **

1. Tenants - does each tenant get its own carbonite schema?

2. Context - I want to associate current_user.id with the Carbonite transaction. However, some operations will be done anonymously

3. Archive - unclear whether anything special is needed

4. Seed data (if applicable). Ash would either insert a transaction for Carbonite or disable the validation. See https://hexdocs.pm/carbonite/Carbonite.html#module-inserting-a-transaction

5. Migrations. Similarly, if migrations alter data, Ash would either insert a transaction for Carbonite or disable the validation. See https://hexdocs.pm/carbonite/Carbonite.html#module-transactions-in-data-migrations

[zimt28]

I haven't looked into Carbonite but have you seen the ash_paper_trail extension?

[zachdaniel]

Carbonite does look quite interesting :) The benefit of carbonite is that it actually tracks all of the changes effectively "automagically" under the hood, allowing for things like arbitrary changes in the db to be tracked even if they don't go through Ash resources. The benefit of ash_paper_trail is that you can track anything that changes about a resource even if that doesn't map to a pg datalayer, and it will be a bit more semantic, i.e we track each "action invocation". They likely have two different places, and I could even see a world where you might use both. From a practical standpoint:

1. tenants - we would likely want people who are using schema based multitenancy to have separate carbonite schemas? I'm not too informed on carbonite to say what is preferred there.
2. Context - This should be quite easy. Since the extension would just add a change to each action, or we may add a new option called something like around_transaction that lives in the api, in either case we can ensure that the callback there has access to the "actor" (user), the "tenant" and any other context.
3. Archive - This should likley just work. All we do for archive is add an attribute and ensure that we filter on it at the application layer
4. Seed data - Hard to say really. We have some utilities for seeding directly to the data layer, so perhaps we can add some kind of config to wrap seeding in a carbonite transaction. Or just some kind of utility like AshCarbonite.transaction("name", fn -> ... end).

I'd be curious to see how this goes!

[terrisgit]

Another benefit of Ash paper trail is json/graphql/etc. APIs provide read-only access to historical versions. Ash could probably support Carbonite similarly but that isn't a use case that I need. Carbonite works with Ecto, so if you have a hybrid app that uses Ash and Ecto, you can standardize on Carbonite for auditing. Carbonite is also mature. For example, it supports partitioning, writing to different schemas, and omitting sensitive columns like passwords. Paper trail can use Carbonite as a guide for must-have features.

[zachdaniel]

you could also derive the type from the action name, i.e type: "#{Ash.Resource.Info.short_name(resource)}_#{changeset.action.name}" to genericize it a bit more.

So the benefit of an extension would probably mostly lie in the fact that it could add custom statements to the migration generator to create the carbonite tables. You'd still have to run the initial migration manually, but then all resources to which you add the extension could be automatically migrated when generating their migrations.

What goes into the add_<<resource>>s_carbonite migration?

[zachdaniel]

You're right though, its a good sign that it was so easy to integrate carbonite 👍🏻

[zachdaniel]

Maybe the folks at carbonate would want to provide this extension I wonder. Its not going to be much code to maintain realistically.

[terrisgit]

@zachdaniel It's a chicken and egg problem. Few people use Carbonite and the Ash community is still small (I will do my part to evangelize and write documentation). I am likely the only person who is motivated to use Ash and Carbonite together. I'm usually in the wilderness. Ironically if it weren't for Carbonite, I would have not appreciated Ash -- as you have said, building nice to haves is hard to justify for any given project. I would build the extension myself if I knew how to make extensions but I am a beginner in both Elixir and Ash. I also think there is expertise in the DSL design that will take me months to acquire. Given the small benefit of the extension, it's probably not worth the effort. A complete solution would address all of Carbonite's needs, as Ash does to hide Ecto -- I don't even need to add Ecto as a dependency. For now I will go with the recipe approach and write some documentation about it when I'm happy with integration. For example, inferring the transaction type is a great suggestion. Also, if each tenant gets its own schema, it would be best to give each tenant its own Carbonite schema using some sort of naming convention. I plan to start with using tenants.

[zachdaniel]

I think ash may have more users than carbonite FWIW. Regardless, I'll add some transaction hooks tonight or tomorrow to make this all work nicely, and then we or I can write up a short bit on integrating them, and just have that as a guide in ash_postgres. I'll put a not to that in the paper trail getting started guide too. :)

[terrisgit]

What goes into the add_<<resource>>s_carbonite migration?

Create and drop trigger statements for the associated table. More thought is needed for many to many relationship tables.

@zachdaniel

[terrisgit]

you could also derive the type from the action name, i.e type: "#{Ash.Resource.Info.short_name(resource)}_#{changeset.action.name}" to genericize it a bit more.

Sorry, I'm new to this. I put that into the resource.ex file but I got:

undefined function resource/0 (there is no such import)

And changeset is also not defined.

I imagine this belongs in carbonite_transaction.ex ?

[zachdaniel]

Yeah, it would go in the Change module