Anchor data on-chain when uploading files and posts

[aaronc]

When we store JSON-LD posts and their attached media files we should anchor the data on-chain using a hot wallet with a minimum balance of REGEN just to be used for anchoring data.

[paul121]

Okay, I've done some research on Graphile Worker and the existing task implementations in regen-server. I have some questions about how we want to implement this, I think we have a few options. But graphile worker should work well. The backoff method for retries should be sufficient, and minus some gas fees, there shouldn't be any harm if an IRI is anchored twice.

First, the task payload and execution.

• The payload should be quite simple, we only need a Regen IRI. All payload are defined json: { "iri": "regen:1234.rdf"}. The task executor can be configured to know anything else it needs to know.
• Ultimately execute the task using the regen cli regen tx data anchor [iri] to build a TX with MsgAnchor, sign using the hot wallet, and broadcast using --broadcast-mode=block to ensure this is included in a block, check response to check it was successful, and return accordingly.
• Question: Once completed, graphile worker doesn't track or store the result of jobs. Do we want to store anything indicating the success of a job, like timestamp or TX hash? This would add more complexity to the task executor, requiring it to execute a query back to the database. This might just be better to do in the indexer.

Scheduling jobs. Two options:

1. Using the graphile worker library addJob() method (currently how the send_email task works).
2. Create a postgres trigger function (example) that uses graphile_wroker.add_job(). Given that our task payload is so simple this could be pretty easy. A trigger could be added to grab IRIs AFTER INSERT ON existing tables: public.upload, public.post, easy to include public.metadata_graph as well.

Executing jobs. Two options:

1. Using an async task executor function (same as current send_email implementation).
2. Use an executable file. This is an exeperimental feature included in the latest release v0.16. Our executable file could be a simple bash script that passes the task payload IRI to regen, does a quick parse on the response, and returns a correct exit code.

Curious what you think @aaronc @blushi. Both option 2s seem quite elegant to me, if you don't mind creating a new function on the DB and testing the experimental feature.

[blushi]

Question: Once completed, graphile worker doesn't track or store the result of jobs. Do we want to store anything indicating the success of a job, like timestamp or TX hash? This would add more complexity to the task executor, requiring it to execute a query back to the database. This might just be better to do in the indexer.

Looking at the designs https://www.figma.com/file/Bksz1JeDYxQVIXdI46EgPT/Project-Posts?node-id=744%3A47640&mode=dev, it seems like we want primarily signing (ie attesting) events (not necessarily anchoring at first) so we want to be able to retrieve timestamp and tx hash for those. I believe that could work indeed with the indexer based on the events https://buf.build/regen/regen-ledger/docs/main:regen.data.v1#regen.data.v1.EventAttest

I believe option 2 makes more sense for scheduling jobs.
Regarding executing jobs, probably worth to discuss during next architecture call.

[paul121]

Regarding executing jobs, probably worth to discuss during next architecture call.

Marie and I discussed this and decided to go ahead and try using an executable file/script + regen binary. Ultimately though we need to make sure we can get the regen binary into the heroku production environment... Although we could maybe deploy a docker image to heroku, it seems like the proper option is to use heroku buildpacks

Buildpacks enable you to extend Heroku's build system to support your language or customizations, or to make particular binary packages available to the runtime.

In particular we'll need to use multiple buildpacks for an app, likely creating a custom buildpack that builds the regen binary and makes it available to the main app. Perhaps this could be useful for other Heroku apps/services for RND?

We should figure out the heroku details before I go much further but I've made an initial implementation of this task executor using the regen binary in the docker container + an executable bash script in #454 . I'll add some more details on this specific implementation in the PR.

[paul121]

We should figure out the heroku details before I go much further

We spoke about this today. There is a little concern about vendor lockin/being too coupled to Heroku if we need to build + maintain a custom buildpack. The alternative is deploying a docker image that might be more general and useful in other hosting platforms.

For now shall we investigate the buildpack a bit more and go with this if it doesn't look too complex to maintain? I believe it does require creating another repo but the buildpack could be quite simple. We may not even need to build the binary if the pre-built release is compatible with the heroku environment. A good example is written here: https://www.petekeen.net/introduction-to-heroku-buildpacks

What do you think @blushi @aaronc @clevinson ?