Capsules dependency management

[antonioribeiro]

Hello everyone!

Not sure if any of you are using Capsules already, but I'm finishing a project at AREA 17 with them (at least one other internal Team is also using it) and it feels good to use it so far. If you need help using it, just mention me here.

Well the next step would be to improve the installer, which exists already but it's not totally ready for prime time yet. I'm working (yes WIP) on 3 different public Capsules now to better understand the needs, and @ifox#4260 came to me one of these days thinking about the --copy option not being the best one for Capsules. So now I'm second guessing @ifox#4260's second guess 😄

For everyone to better understand the what we are thinking, we have two installing options:

• Installing with --copy (implemented, but miss a lot of things)
• Installing with --require (not implemented)

Currently the COPY command just downloads the zip and unzip it on your /Capsules directory. Implementation is very rudimentary, but it's gettind the job done. But I believe we still need it to do more:

• Have a dependency management (the public Homepages Capsule depends on Base, as Cities Capsule depends on Countries and Base). A PR was sent and this is also working fairly good. I did a PR for this: Add dependency resolver to twill:capsule:install #933.
• A way to rename the Capsule while installing it: sometimes we need the same behaviour with a different name, but Capsules (modules) contains a lot of classes and files, manually renaming them now is easier because everything is on a single directory, but it still take some time and effort.
• We won't get any updates for the Capsule, is this a problem?

The REQUIRE command would, in other hand, use Composer as a dependency manager, which is, of course, fantastic, but we have some things to think about here:

• Are we installing Capsules on the vendor folder?
• We can probably tell Composer to put them on a app/Twill/Capsules, as, I believe, Drupal does.
• And if I want to make changes to the Capsule, should I be forced to fork it on Github and change something small only for that particular project?
• Or should I just delete the .git repository, remove it from the composer.json and commit it definitely on my project?
• If the Capsules maintainer changes something on the Capsule, my project can break, right? Should we care about this? This is how package dependency system works after all.
• Do we really need Capsules to be upgradable? Why?
• A Capsule that has a composer.json, can have all sort of dependencies (packages and other Capsules). We probably want packages to get updates, but we still want Capsules to receive changes?

I know I'm asking the same question over and over, using different words, but I feel that Capsules are so connected to the project we are building now, because as soon as you install a Capsule you may want to add or remove fields from it, or change the order of the fields in the form, or just disable translations because your project has only one language. So I feel that they are too much of a living being and cannot do everything, for everyone, so that "require" option could pose more problems than solutions.

Unless we keep the dependency management for packages and disable it for the Capsule itself and just let users to modify them if they want.

What do you think about all this?

Cheers!

[pboivin]

Hey @antonioribeiro!

A lot to think about here :)

I haven't started using Capsules yet, but I really see the appeal!

I think you put the problem in context quite well. From my point of view, the first question that comes to mind reading your message is: Are Capsules intended as "boilerplate" to be customized and extended, or more as "plugins" to be integrated and built around?

At a glance, it looks like --copy supports the "boilerplate" approach, whereas --require supports the "plugin" approach. Does that make any sense? I think that both use cases are likely to happen in the wild.

I agree that dependency management is a must. For what it's worth, I also think that a line in a README that says "to use this Capsule, you must run composer require xyz" is pretty okay as a form of basic dependency management...

I really see that the question of why and how to update a Capsule is at the heart of the problem. I don't see how Capsules as "boilerplate" can be updated — once it's in your project, it's yours. On the other hand, Capsules as "plugins" can very well be updated, whether via composer or just unzipping, really.

Not sure if I've actually added new thoughts to the discussion at this point but, I'm interested in the subject ;)

[kirkbushell]

Using composer to manage the capsules and a dependencies is imho, a no-brainer. It offers a wealth of opportunities and advantages over manually doing this. That said, publicly available capsules should be configurable just like the rest of the system, rather than being copied or "published" to your local project (I see this introducing a world of problems, the main one being, as you suggested - capsule updates).

They should be viewed as mini repositories with a dependency on Twill versions.

[pboivin]

@kirkbushell I agree with you on the model of Capsules as mini repositories. Although, I also see how — a Capsule being made of "Twill stuff" (migrations, models, views, ...) — there could be a point where configuration options just aren't enough. As @antonioribeiro mentioned, when working with a Capsule, I may want to reorder fields, add some extra processing over here, update the styling over there, etc. At some point in this process, it's just easier to --copy or fork than to configure. Does that make any sense?

[ibes]

My background:
Haven’t used Capsules. Coming from WordPress.

I see use cases for both require and copy.

But first of all:
Telling composer to install certain packages in certain places is not difficult. The main composer.json needs to know the rules and the packages should have a certain type to be recognizable for the rule.
I know that from WP with composer setups.

In WP there is the rule to never touch a plugin directly to not break my own modifications by updating the plugin.
To still be able to modify the whole hook system is needed. Filters, Actions.
That works ok.

I see a use case for --require mostly in high optimized or specialized stuff.
SEO or media handling. Where I want to get delivered with the hottest updates.

I see --copy more where I want a boilerplate with best practice that I can quickly delete down to what I need and customize.
A “customer feedback” content type, a blog setup, a frontend auth handling

Maybe capsules need to be split in two things?
Maybe it is cool to just give people both options so they can go for what works best.

Could there be a mix of boilerplate and updatable parts.
Having the most important parts in a core capsule that is expended by the boilterplate capsule.
Is that the “publish” mechanism of Laravel?!

I guess for trivial stuff I would just want to get a head start.
For difficult things I would want to get the most difficult/complex things updated while being able to adjust it to my setup.

[antonioribeiro]

Thanks for your feedback @pboi20 , @kirkbushell , @ibes!

Looks like REQUIRE is a must, so let's try to do this, and see how it plays. I believe it could work well for really big, super complex, or highly configurable Capsules, where people would not feel the need to change as soon as they touch it.

@ibes , you came up with a third kind of capsule: HYBRID, nice one! But the way Twill modules (and Laravel) are made, a very simple update (like adding a new field) require changes to at least migration and model, if you are adding a relationship, you may have to change the repository, if the index or browser are customized, you may also have to change the controller. And I'm just oversimplifying it, because Twill is huge and pretty much anything could require a change to a class, which would prevent the Capsule from receive updates. But I would love to hear more about what could be the implementation for this.

Maybe we can, in the future, make Capsules less dependable of some classes, relying more in configuration (I'm more a configuration over convention guy), using DTOs or configuration objects (as pointed by @kirkbushell on #internals @ our Discord), but I feel we have too much unneeded boilerplate code. For example, seems to me that we don't really need this kind of controller, and I have many of those:

<?php

namespace App\Twill\Capsules\Collections\Http\Controllers;

use A17\Twill\Http\Controllers\Admin\ModuleController;

class CollectionController extends ModuleController
{
    protected $moduleName = 'collections';

    protected $indexColumns = [
        'title' => [
            'title' => 'Title',
            'field' => 'title',
        ],

        'city' => [
            'title' => 'City',
            'field' => 'city_name',
        ],
    ];
}

We can probably have a configuration for the index and a more global Controller class handling the requests. Same for migrations, we could probably use an object like Laravel's validation Rule to define migrations, same for Repositories (if we are only adding afterSave() and getFormFields() to handle browsers, repeaters, relationships, but models in the other hand looks harder.

Laravel Nova also uses it extensively, that's probably the cleverest part of it, and it's also super powerful:

public function fields(NovaRequest $request)
{
    return [
        Text::make('First Name'),
        Text::make('Last Name'),
        Text::make('Job Title'),
    ];
}

We could possibly do forms this way, like Nova too. But Twill forms are also a different kind of beast.

If we are able to do something like this, maybe we can have HYBRID Capsules, configurable and upgradable. But this is how I see it.

What about you?

[kirkbushell]

The only difference between Twill forms and Nova forms (other than configuration) is that Twill has the block editor (blockable) fields. As long as that is taken into account when developing the solution, it would be ideal.

The way I see it currently, is that if I could use the block editor in Nova, that would be my go-to, every single time. Nova does everything better, except content. Twill imho is a leader in that space, and combining the projects or even supporting Nova through a paid Twill package (I would HAPPILY pay for a block editor plugin for Nova!!!)

[antonioribeiro]

I can tell more things Twill does a lot better than Nova:

• Image/File management
• Image cropping: this is something I didn't see anywhere else, not the way it was implemented on Twill.
• In form translations (does Nova even have translations?)
• Preview: can you preview your frontend inside Nova, and view pages exactly the way your users will see them?
• Revisions: can you visually compare (preview again) two pages and choose to rollback to an older one?
• Easily publish/unpublish records. Maybe we can have this with some plugin in Nova, adding it automatically to all forms, but Twill's implementation is also beautiful and has a great UX, and that counts too, right? 😜

[kirkbushell]

Yup, all fair points :)

In all honesty though - the rest to me doesn't matter. Just give me the content block editor. lol

Revisions are actually really nice, I reckon that could be written as a plugin as well. Translations are a much more difficult problem to solve.

Also - Nova isn't mobile-friendly at all.

[slimko]

@kirkbushell free block editor for Nova https://github.com/whitecube/nova-flexible-content

[kirkbushell]

@kirkbushell free block editor for Nova https://github.com/whitecube/nova-flexible-content

I've ended up using this now in several projects. Thank you :)

[antonioribeiro]

Hello everyone, my first take on making Composer install Capsules into our own Capsules folder failed, because I failed to understand the concept of a project in Composer.

As most of you know, Composer allows frameworks to configure installer-paths, what I didn't know when I first tried this is that this feature is tied to a project, so what's the project? It's the main Framework on what your application is based on. In our case this Framework is Laravel, not the Laravel Framework package, but laravel/laravel. In the case of Drupal and Wordpress, it's of course Drupal and Wordpress main repos, and no a third party package like Twill. There a list of the supported ones here.

I believe (yes I'm still not totally sure) that in order for this to work our application composer.json would have to be based on a "project" composer.json which should look something like this:

{
    "name": "twill/project",
    "type": "project",
    "require": {
        "php": "^8.0",
        "area17/twill": "^2.5.1",
        "area17/twill-capsule-homepages": "^1.0",
        "laravel/framework": "^8.54"
    },
    "extra": {
        "installer-paths": {
            "app/Twill/Capsules/Vendor/{$name}": ["type:twill-capsule"]
        }
    }
}

This is not impossible, of course, but this means a lot of changes to the Twill ecosystem, like creating a "Twill Project" repository, based on the laravel/laravel, of course, possibly an installer and who knows what more...

Any ideas on how could we tackle this without going down this path?

Cheers!

[pboivin]

@antonioribeiro I appreciate all the thoughts you've been putting into this problem! This all makes sense to me, especially the need to --copy for bespoke applications.

If I may just share a minor nitpick:

I disagree on the need to differentiate between plugins and capsules. I think there's a lot of overlap conceptually, and the distinction is really mostly philosophical. Say I'm developing a "plugin" that needs to hook into Twill internals as well as: install views, models, repositories, a CMS menu entry... What if I find a way to wrap that in a capsule as a means of distribution?

I guess what I'm saying is mostly — We don't need to introduce a "plugin" entity into the mix. I believe vanilla laravel packages already fit the bill. We may just need more (a lot more) "hooks" to extend what is possible.

[kirkbushell]

@pboivin Actually there is a point of difference between the two. I would think that all capsules are plugins, but not all plugins are capsules. Ie. Maybe a plugin allows you to change how user registrations are done (socal auth, perhaps?), but a capsule is a full CRUD "plugin" installed, maybe like a comments capsule?

There is certainly a difference between the two, but both should behave from a dependency perspective in the same way.

[pboivin]

@kirkbushell I fully agree and I believe we are arriving at the same conclusion – Capsules are a kind of plugin and nothing new is needed to support other kinds of plugins. Mostly "hooks" to extend Twill core.

[kirkbushell]

@kirkbushell I fully agree and I believe we are arriving at the same conclusion – Capsules are a kind of plugin and nothing new is needed to support other kinds of plugins.

Quite the contrary, quite a lot is needed to support a decent plugin architecture for Twill :) But I believe Capsules should be an extension of that system, and is really where the work should begin.

[freshleafmedia]

While it doesn't address the plugin problem I've had a crack at making Capsules Composer installable: #1291

They can co-exist with Capsules in the app/Twill/Capsules directory so this allows users to have their own bespoke Capsules while also including common Capsules from Packagist

[antonioribeiro]

I know we can do anything with code, the question we have to pose ourselves is: should we?

Just to scratch the surface of a plugin system for Twill here, I believe a real plugin system for Twill would have to have support for VueJS. So are we adding JavasScript compilation to Modules now? Because this is clearly something that Modules doesn't support today, and, as said before Capsules are nothing more than Modules living on their own directory tree. And if we don't do this, well, Capsules will also start to become incompatible with Modules, or maybe just do more things than Modules do and, but this can be very confusing.

[kirkbushell]

Yup, and Vue JS doesn't play very nicely at all with a decent plugin architecture - one of it's biggest flaws.

Laravel Nova handles this rather well, imho. There could be some lessons there that we could take onboard.