Packages & Codegen

[brudil]

I'll just preface this by saying this is really only for discussion, it's not entirely joist scope here.

We've been reworking our application packaging. One of the great things about Rails and Django is the ability for gems/apps to utilise the their orm. Things like ActionStore and django-notifications-hq.

Obviously there's a ton of stuff outside of Joist here, but the nature of codegen within Joist does make this more tricky.

Just curious if there's been any thoughts/patterns around this. I've got a couple of ideas to play around with, largely tied to our IaC container, but nothing that feels right just yet.

Pure ideation here

[stephenh]

Hi @brudil !

I've only done a few ~6 months stints with Rails/AR, so don't deeply understand the rails gem ecosystem, but agree that in principle the Joist/ORM entities should be usable in a generic way.

What specifically about codegen makes this hard?

I guess looking at https://github.com/buhrmi/actionstore, they are adding methods to the entity, i.e. author.push_append(data), by doing the has_actionstore thing?

Which, yeah, not sure that is a Joist limitation per se, and more that TypeScript doen't have that sort of magic yet (microsoft/TypeScript#4881. :-(

I see you're prototyping the extends FooMixin(Codegen) pattern--fwiw I've never really gotten the extends FooMixin pattern to work as well as I want, but I'm kind of forgetting why. Would be great to here if it is/is not working well for you.

Personally, I would probably just not add the methods to the entities themselves, but do something like getActionStore(author) that takes an entity and returns an ActionStoreWrapper<T extends Entity> that wraps the author and adds the push_append type methods.

Googling around a bit, the internet suggestions that something like:

type Constructor<T = {}> = new (...args: any[]) => T;

interface ActionStore {
  pushAppend(): void;
}

function ActionStore<TBase extends Constructor>(base: TBase, context: ClassDecoratorContext) {
  return class extends base {
    pushAppend = () => console.log("New method added!");
  };
}

@ActionStore
class MyClass {}

interface MyClass extends ActionStore {}

const c = new MyClass();
c.pushAppend();

With the new TC39 stage 3 decorators--the only con being that just @ActionStore is not enough to get the type updated, and so the interface MyClass extends ActionStore is also necessary.

But, dunno, those are my two initial ideas (a getActionStore helper/wrapper method, and the decorator+interface combo)...

I noticed your diagram also mentioned "NotificationCodegen not included in the build"--are you trying to ship the Joist types to the frontend?

We don't have our production FE code using the Joist domain types (it uses GraphQL), but we have been putting the *.d.ts files from our entities into a types-only npm package:

mkdir -p ./dist/entities/factories
mkdir -p ./dist/entities/codegen
mkdir -p ./dist/entities/enums
cp ../dist/src/entities/*.d.ts ./dist/entities
cp ../dist/src/entities/factories/*.d.ts ./dist/entities/factories
cp ../dist/src/entities/codegen/*.d.ts ./dist/entities/codegen
cp ../dist/src/entities/enums/*.d.ts ./dist/entities/enums

And then using those on the frontend to write "fixtures" that are just *.ts files that import newAuthor from our-service-factories npm package, and then instead of actually invoking the factory script directly on the FE, we ask the BE to eval it, which basically runs the factory code, and lets the FE's use the BE factories to setup their test scenarios (of course only used for local development).

[brudil]

Appreciate the response, apologies for that slightly terse rambling dump of thoughts. I think I might have failed to explain the intent/idea here.

The gist is

How far could a backend framework which uses Joist as it's blessed built-in orm go with producing externalised apps.

I'll be honest I've way more experience in Django than rails, so I might have confused things referencing ActionStore which I haven't used.

Django as a framework consists of 'apps' - these contain the cross section of a feature of your app. I'll expand on an example from the django docs;

my_project/
├── manage.py
├── my_project/
│   ├── __init__.py
│   ├── asgi.py
│   ├─ settings.py
│   ├─ urls.py
│   ├── wsgi.py
├── my_app/
    ├── migrations/
    │   └── __init__.py
    ├── __init__.py
    ├── admin.py
    ├── apps.py
    ├── models.py
    ├── tests.py
    ├── urls.py
    └── views.py

A django project contains multiple apps which fit this shape, say a Todo project might have project, task, reminders, badges etc apps.

The bit that I'm curious in is the fact that Django Apps can be external packages. This is really nice because it means you can use django-notifications and have a Notification Model that you import from that package and can use! And it's more or less turn-key. I can take badges app, and as long as it doesn't have dependacies on other models outside of it, and chuck it up as a external package and it works.

This isn't just great for open sourcing reusable apps, but modularising internal apps.

Obviously there's a fair bit of frameworkly stuff going on here, handling migrations etc which is out of scope for Joist.

For orms like Django's which is entirely class based, the external Notification Model is a class using the Django API, hand written;

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

So as long as the externalised model is compatible with the Django version it's used with, you can just import it and go.

---

Where I see this breaking down in a joist world.

I'll use Notifications as an example; let's say in this magical joist-first framework which supports Django-like apps. I'll call it Zest.

So I want to use zest-notifications which provides a some nice business logic + an entity + and db migration. The business logic wants to utilise the Joist Notification entity that it provides, but...

Joist needs a schema to generate from. And codegen is entirely coupled to each joist version (i.e don't expect codegen output from 1.1.x to work with joist-orm 1.1.y. We also want zest-notifications to be independantly testable.

This leads to the following assumptions:

• The root project has to be the thing creating NotificationCodegen - a single joist codegen run looking at the final schema produces the *Codegen files used everywhere.
  • This significantly raises version match issue as the external joist API changes less that the internal codegen<>orm one.
  • It also means that the notification defined within the external package can be enhanced (additional columns etc) via the db schema and relations will just work with root project entities.

So there's a couple of goals here:

• Have a working joist entity within zest-notifications
  • Needed for testing (its own generated NotificationCodegen)
  • And for typing
• When using it in a root project, we can utilise the root project codegen

This is what my first idea was (badly) attempting to denote.

This is where the mixin idea came in.

zest-notification might define a migration that looks like

createTable('notification').addColumn('id', 'pk').addColumn('read_at', 'timestampz')

The user-land class - which extends from the codegen class need to support both the external codegen (types + tests), and the root project codegen.

As a super simple example;

This would be the output in a normal joist app:

class Notification extends NotificationCodegen {
  get isRead() { return this.readAt !== undefined }
}

My line of solutionising was thinking about how we can make the codegen extends dynamic.

In the external package

// NotificationMixin.ts
function NotificationMixin(codegen) {
  return class extends codegen {
    get isRead() { return this.readAt !== undefined }
  }
}

In the root project, we would be able to utilise the minxin

class Notification extends NotificationMixin(NotificationCodegen) {
  // and custom stuff the root project cares about that is not known to the external app.
}

Then the external app, is provided with the root project Notification class to use at runtime, but it's complie time types utilise its own *Codegen file.

This doesn't touch upon the frontend, but external modules exposing entities which they utilise themselves.

Hopefully I've cleared the problem space up a bit. If you couldn't care less about this, please don't feel the need to spend time thinking about this - I'm just playing around here because it's what I think is an interesting problem to solve.

[stephenh]

Ah I got it; I have actually thought of similar ideas, i.e. shipping a "comments" module or "attachments" module that, if you include these tables in your schema, a Comment entity or Attachment entity shows up in your domain model, and now you can easily say "my author has comments" or "my book has attachments".

Is that a good articulation of the use case?

Like:

• the app/module controls its own db schema
• the app/module hence gets "its own entity"
• but if we package up the zest-comments as an NPM package, then who creates Comment.ts & CommentCodegen.ts?
• in theory zest-comments wants to own/ship Comment.ts b/c it will have custom business logic/validation rules/etc. within it
• but at least out of the box, joist-codegen will see the comments table and be like "I'm making a Comment.ts/CommentCodegen.ts np!" but then they stomp on each other

I agree it would be really neat to support this sort of setup, and I'm interested in making it happen.

Fwiw besides annotations & wrapper functions, I did forget another option: codegen plugins. I.e. we already have a joist-graphql-codegen plugin that admittedly doesn't touch the entities themselves, but just does further scaffolding of GraphQL artifacts based on the schema.

So, dunno, maybe we could have in joist-codegen.json a plugins: [zest-comment-codegen] line that maybe ... says "hey I'm claiming the comments table, and here's my npm-package-provided Comment.ts for it"?

As you said (and assuming I'm following the use case correctly), we'd need to make sure the comments-table-in-the-users-database matched the Comment/Codegen.ts-from-the-npm-package 1:1, not just in fields/columns, but right also versions of Joist.

Just given how coupled the CommentCodegen.ts files are to a specific version of Joist, maybe your point was that the zest-comments npm package should have just the Comment.ts business logic, and we let the local project go ahead and scaffold out the CommentCodegen on it's own. 🤔

Am I following the use case so far/well enough?

[brudil]

Yeah, I think we're on the same page.

If we take it back from implementation details, and consider the holy grail;

Say I have a system that say handles attachments, and I want to make it reusable across multiple projects. This package is able to provide:

• A base db schema
  • This is OOS for Joist
• Entity enhancements
  • Essentialy its Entity.ts
• The entity is usable within the package, via the em, with the scope of the package avaiable.

So for example;

framework-mailer

// migrations.ts - joist doesn't care about this, but might be part of the framework, or it's part of the documents "framework-attachments" expects x table, with at least x, y columns.

export migrations = [knexmigration, knexmigration];

// Attachment.ts

class MailRecord extends ?? {
  // holds domain logic for the entity

  // methods
  loadBuffer() { return FileAdapter.from(this).loadBuffer() }

  // getters
  get size() { return FileSize.fromKB(this.bytes) }

  // joist api too, such as reactive field etc
}

// config api too

// logic.ts - this is frameworky but to demostrate how a package could use the entity too

export function sendMail(mailer) {
  // send mail etc
  em.create(MailRecord, mailer);
}

// route.ts - this is frameworky but to demostrate how a package could use the entity too

export rotues = [
  {
    path: 'webhook/mailer/ingest',
    handler: (req, reply) => {
      const mail = await em.load(MailRecord, req.body.mailId);
      
      mail.status = req.body.status;

      return reply.code(200);
   }
}
];

Hopefully I haven't over complicated thing here, but imo the challenge is utilising the project codegen file for consumers, but generating another for package authors, for tests and typing. The nature of classes makes this hard, hence codegen, mixins or some other magic might be required.

[stephenh]

Yeah, it's an interesting problem; I could also see the project-codegen file have it's own unique aspects, i.e. something like:

• zest-comments / src / CommentCodegen.ts has a `parent: PolymorphicReference<Comment, CommentParent>
• zest-comments / src / Comment.ts: has been logic in it
• ideally ^ are shipped in an npm package?
• main-project / src / CommentCodegen.ts has the same parent: PR<Comment, CommentParent> but now we know the main-project actually has ~5 specific CommentParents, i.e. it attached the comment to a Book, Author, and BookReview
• main-project / src / Comment.ts: might want some main-project specific logic as well

I wonder if we could use/abuse inheritance to do something like:

• main-project/src/Comment extends main-project/codegen/AppComment.ts
  • The codegen/AppComment.ts is basically a copy/paste of the zest-comments/src/Comment.ts but into a "read-only" file that acts as the base class of Comment.ts but also extends the project's CommentCodegen instead of the zest-comments CommentCodegen.ts
• `main-project/codegen/AppComment extends main-project/codegen/CommentCodegen.ts

This is admittedly pretty weird, but we're basically trying to:

1. Use the main project's CommentCodegen.ts as the source-of-truth for what our database actually looks like
2. Somehow insert the zest-comments/app's layer of business logic from it's Comment.ts "on top" of what our database actually looks like, and
3. Provide our own main-project/Comments. that is both of ^ glued together

I dunno, I guess do you want the main-project to have "its own Comment.ts/CommentCodegen.ts"? To have main-specific logic, or other "changes introduced by the main be schema"?

I could see the answer going both ways (which maybe means they're both valid, but separate, features?), i.e.:

1. Yes, main-project having its own Comment/CommentCodegen lets use "inject" the zest-comments functionality very quickly, and still make our own adjustments (in particular hook it up to other entities in our domain)
2. No, we want the zest-comment/app entities to be more like "modular monolith modules" where they are more isolated, and if zest-comments/app/Comment "owns" the comment entity, we actually don't want it showing up in our main-project domain model at all.

[brudil]

Moving the package's entity file (zest-comment/Comment.ts) to the project's codegen creates a few concerns for me around special semantics for module scoping;

Comment.ts might

• import symbols from the package which are not exported
• import symbols from it's own dependencies

Perhaps it's the best way forward, but it definitely would add a fair bit of friction to package authors and consumers which would be better to avoid.

Latter question is an interesting one. I can see it being used for both cases. I've imagined a "subset" stratergy - essentially inline with TypeScript's structual typing, where the external package should be able to operate on the schema it defines, but if there is additional domain/schema defined on the entity by the main project, it wouldn't cause issues and is usable by the main project.

This could be very powerful, where a external package could expect logic to be defined by the project.

I've started to consider some ideas around codegen for just types which I'll contine to play around with.