Planned refactoring and temporary feature freeze

[ganigeorgiev]

The last couple weeks I've tried experimenting with slightly different internal structure and decided that it is time to share some notes regarding the planned refactoring and hopefully this will bring some clarity in what direction the project is headed.

The main takeaway is that after PocketBase v0.22.0 release there will be a temporary feature freeze for some time until I manage to finalize the refactoring.

Please note that there is no fixed timeline when it will be completed but I'm hoping that we will have at least a v1 alpha release with pretty much finalized APIs by the end of this year (this is also the limit to how much I could stretch my current sabbatical and will have to start looking for a regular employment).

With that said, below you can find some of the more notable planned changes:

Merge Dao with core.App

There are a lot of scenarios where passing daos.Dao is not sufficient enough, for example you may want access to the app settings from a migration, or the app configured file system, or to trigger manually a hook, to send an email, to access the logger, etc.

By eliminating the separate Dao abstraction and moving its logic to the core.App we can simplify the app related operations and have only 1 common interface that you can pass around (including when dealing with transactions).

Additionally we will remove the various Dao.Save(m), Dao.SaveCollection(m), Dao.SaveRecord(m), Dao.SaveAdmin(m), etc. methods and will replace them with App.Save(m) (and App.RawSave(m) when you want to skip validations).

Remove/hide the RecordUpsert (and other similar forms)

Currently there are 2 main ways of modifying records:

• the forms.RecordUpsert model for when you need validation
• direct dao.SaveRecord() call for when you want a direct db insert

We can simplify all of this by avoiding the extra form abstraction and move the form validation in the Record model itself.

The end user will operates only with Record and custom validations and anything field specific will be done through it field structs.

For example, to upload a new file programmatically, currently we have to do something like:

// with forms.RecordUpsert
// ---
form := forms.NewRecordUpsert(app, record)
form.AddFiles("fileField", file)
form.Submit()

// or raw Record model update
// ---
// do some validations eventually....

// upload the file to the configured app storage
fileKey := path.Join(record.BaseFilePath(), file.Name)
fsys, _ := app.NewFilesystem()
defer fsys.Close()
fsys.UploadFile(file, fileKey)

// save the Record model field with just the uploaded file name
record.Set("fileField", file.Name)
app.Dao().SaveRecord(record)

We can replace both options with just:

record.Set("fileField", file) // just like any other regular field value (record.Get("fileField") will return a File instance)
app.Save(record) // or RawSave() if we don't want validations

This would also simplify removing existing or intercepting uploaded files since you can just remove them from the Record instance like a regular field value and call app.Save(record) to persist the changes.

Replace echo with a thin wrapper around the new Go 1.22 mux enhancements

Go 1.22 introduced some enhancements to the builtin HTTP router which we can try to make use of it in order to minimize the external dependencies and allow broader compatibility with the standard Go library.

For routing we currently rely on echo (v5_alpha) but we don't really use that much of it since we have to supply our own custom error handling, Binder, Serializer, etc. implementations anyway to accommodate better some PocketBase specific use cases.
Additionally I've also seems to have made a wrong assumption that the echo v5_alpha branch was going to be released soon after the first PocketBase release back in 2022, which in hindsight based on current echo development pace and some recent discussions it doesn't seem to be the case anymore (see also labstack/echo#2000 (comment)).

Rewrite the Admin UI to be extendable

At the moment we support extending some of the PocketBase server-side functionality with Go (or JS pb_hooks), but there is no support for customizing the Admin UI, which prevents us to offer full-featured "plugin"-like functionality.

To make the Admin UI customizable we will have to:

• Create a small standalone HTML/CSS components library (buttons, inputs, alerts, dialogs, icons, etc.) to have a more cohesive design with the custom users UI.

• Replace the current Svelte SPA with a framework that doesn't require a JavaScript build step.
  I haven't decided on what we'll use but most likely it will be either Vue.js or Alpine.js (the Admin UI will still be a plain static browser application).

• Provide client-side hooks/injection-points for registering custom user content (menu items, custom collection fields, custom app settings, etc.).

Summary

There are a lot of other smaller changes that will be introduced together with the refactoring:

• support for user defined SQLite build (which as a side effect will allow connecting to other databases that supports the standard Go database/sql interface but for the time being I'm not planning officially supporting other SQL dialects)
• add option to customize the default system fields (id, email, username, etc.)
• support for custom collection fields
• slightly different records hooks execution order
• attempt to fix the currently on hold OAuth2-email operation edge cases
• etc.

While most of the planned changes may look cosmetic on the surface, they require a lot of intertwined modifications and practically rewriting most of the existing tests, which will take some time.

The short-term plan for now is:

• Release PocketBase v0.22.0 with the latest changes from the develop branch (+ eventually maybe the back/indirect relation filtering), in the next couple weeks. This will be the last major release before the refactoring, meaning that there will be temporary feature freeze.

• Start working on finalizing the refactoring of the server-side changes and rewrite the necessary tests.

• Add migration guide and update the docs to reflect the new changes

• Create a HTML/CSS UI components library, rewrite the Admin UI with Vue.js/Alpine.js and add support for client-side hooks.

Feel free to share your suggestions below if you have something else in mind. I may not reply immediately on it but rest assured that I'll take it in consideration during the refactoring.

Regards,
Gani

[pnmcosta]

Not to bandwagon on hype, but I think that if you're considering replacing the admin with another framework might be worth having a look at HTMX.

I also have recently come across a use case where read-only fields, e.g. editable to create, but disabled for update would be really handy, I've had to use the update events to ensure the current field value is always set with the original record, if this could be added it would be really neat.

Specifically my use case is for a migration seeded collection of templates, I have sanitize and fields option that aren't really modifiable, but the value for the template is.

Thanks for all the work and looking forward to the new version(s) 👍

[dschissler]

I agree with HTMX, but it also doesn't have to be only that. It pairs well with other buildless frameworks.

[rlnorthcutt]

+1 on HTMX especially if you are considering AlpineJS. Go + HTMX is a great combo, and this should really simplify the entire system (performance will be great). This approach would likely also make it easier to build out admin plugins. Users could always add in Svelte/Vue/Js/etc if they want on top.

[vipas84]

HTMX + Alpin

[gedw99]

I and many others are already using pocketable with HTMX. I think there are about 50 to 100 projects on GitHub doing it this way.

It's pretty easy and nice. V2 HTMX is out in beta and really makes it even more modular.

Testing / regression testing can be done with just golang tests that hit a Browser.
Testing of the DAO layer is already easy.
Testing of Notifications that fire off the DAO layer requires a little more thought though.

I would really like to extend the Code gen off the DB for golang too, so that DEV and TESTING is easier and tighter too.

The Pocketbase Admin GUI will be much easier to extend then I think too.
Would be happy to help on this.

The new go 1.22 Router is great and will make maintenance of Pocketbase easier, as les updates. Perf is good enough.

Is there a Branch to look at @ganigeorgiev ? If not, happy to help in any way I can.

[paoloricciuti]

Replace the current Svelte SPA with a framework that doesn't require a JavaScript build step.
I haven't decided on what we'll use but most likely it will be either Vue.js or Alpine.js (the Admin UI will still be a plain static browser application).

Just out of curiosity: what kind of extensibility you are looking for? And why the presence of a compiler would stop the users from extending PocketBase? Willing to contribute to find a solution to this which doesn't require a rewrite...i personally think it's doable. 🙂

[benmccann]

Yes, each job has a tool that's a best fit for it and one tool shouldn't be used for every job, so no worries if there's something that ends up being more helpful for this specific use case than Svelte. I just wanted to understand the requirements and share ideas

Yes, we can allow injecting plain html but we will lose the reactivity and templating language (and the XSS safety that comes with it)

This is true in Svelte 4. In Svelte 5, you wouldn't be forced to lose reactivity since the reactivity system is moving from compile-time reactivity to runtime reactivity. For setting values, users can use the appropriate APIs like .textContext to mitigate XSS risk.

I just don't see an easy way to load plugins during runtime and avoid the extra build step

Here's a dumb example of a plain JS plugin loaded in a Svelte file. I agree if you want to load a .svelte file without having authors first compile it that would be difficult

If users want their plugin to be plain HTML/JS nothing is stopping them from doing that.

I guess it's just a question of what the first class citizen is. What will examples and documention show? You could fully support multiple paths, but that of course comes with its own maintenance burden

[rl-ico]

AlpineJS + HTMX could be the way to go.
No build step necessary.
Works very well with GO.

[whmeitzler]

Speaking as someone who knows JS, but not the current crop of fronted technologies:
Solving this with a specific web technology will drag everyone into that particular implementations idiosyncrasies, build systems, runes, and twitter flame wars.

Is there a way to bind a JS object/interface into an imported web component? Such that the imported web component could have access to hooks into the core admin ui?

There's something very elegant about the hooks architecture of core pocketbase. Maybe we could view the core admin ui as a piece of software whose behavior can be modified by hooks using the same thought process? So, a ui plugin loads, registers its hooks, and does whatever it wants within the component?

I feel like there's an elegant solution hiding in web components. Somehow.

[rl-ico]

Web components are wonderful for sharing encapsulated functionality.
I've built whole design system libraries using web components and while it has definitely a lot of advantages we learnt also all the pain and drawbacks that come with them. Leading us to ultimately recognize that Web components are great to deliver things that have everything inside of them and you want to share it with someone else.

Building web components by hand is very tedious and can be painful. Sure, you have full control, but yeah, full control leads to full amount of things to look for and to build in order to create what you want to create.
This leads to the need of using a framework like stencil or Lit which then will add the need of a build step...

I love web components. I really do. But they really need to be used for what they were designed for. sharing self enclosed functionality.

That being said: I mentioned AlpineJS since it alleviates a lot of things that would take longer to accomplish writing vanilla.
I don't see it as a must and I agree with you about using the Platform instead of solving problems with libraries creating dependencies which could cause problems down the road.

So we could let Alpine out and write a small library for all this little things that makes frontend development more pleasant and efficient. and use HTMX.

True, HTMX is also JS but it gets out of the way and I as a platform purist, can tell HTMX is what HTML should be and I strongly believe that it will become part of the specification at some point.

Of course it is not my decision to make and I am the last person to get attached to something. Just sharing my thoughts on how I think the challenge at hand could be solved the simplest and most efficient way possible meeting the main requirement of not having to have a build step.

Cheers!
R

[martinklepsch]

I for one think the Admin UI is great and I think instead of a full rebuild I think a UI extension point could perhaps be designed outside of the existing admin UI. Of course some helpers to quickly build simple UIs (tables, forms) would be very handy, but IMHO they don't need to live inside the existing admin UI.

I'm partly saying this because I can see this being a huge project that may hav a long turnaround time and could stall momentum in the community but also for you as the author.

[hungcrush]

Sorry. Is Pocketbase stopping development at version 0.22.0 until the v1 alpha release?

[ganigeorgiev]

No. There will be a feature freeze until the refactoring is complete.

[dschissler]

Just today in the link below I learned about Ponys HTML element library. It's only 1KB minified. It seems really flexible. It allows at least 3 ways to wrap template literals into shadow dom. You can put your HTML, CSS and JS into a single file, or an inline template tag or via a string. So it makes single file custom HTML element web components.

I'm thinking that you could offer a combination of built-in libraries that are complimentary to each other. So that means HTMX is a must. Then you could offer some variation of Lit, Alpine.js and what seems pretty nice (at 1KB pre-zip) is Ponys.

Another way of saying it is that HTMX is great because it works with everything to make life easier in some cool ways. Alpine.js is pretty great for it's simplicity and Golang template and HTMX synergy. A Custom HTML component library really compliments Alpine and HTMX very well because they're just HTML tags, and those libraries are all very super tiny.

Alpine on its own is kind of rough because you have to duplicate important markup and script again each time that you want to use it. The hard parts of that should be put into web components.

As you might vaguely recall, I recently took a sabbatical of about 10 days while I was playing with the Pocketbase GOJO JS side of things and how it worked with Alpine, custom element, Golang templates and others. So I feel pretty confident about my take on this.

import Ponys from './ponys.js';
Ponys.import('modal-container', './components/modal-container.html');

<!-- modal-container.html -->

<div
	id="container"
	onclick="event.stopPropagation(); host.close()"
	onkeydown="event.stopPropagation(); if (event.key === 'Escape') host.close()"
>
	<article
		id="modal"
		part="modal"
		aria-modal="true"
		tabindex="-1"
		onclick="event.stopPropagation()"
	>

		<slot></slot>

		<div
			id="controls"
			part="controls"
		>
			<button type="button"
				title="Close"
				onclick="host.close()"
			>
				<span aria-hidden="true">&times;</span>
			</button>
		</div>
	</article>
</div>

<script>
	export default class extends HTMLElement {

		close() {
			this.hidden = true;
		}

	}
</script>

<style>

	:host {
		z-index: 9999;
		position: fixed;
		top: 0;
		left: 0;
		bottom: 0;
		right: 0;
		background: rgba(0, 0, 0, 0.5);
	}

	#container {
		display: flex;
		flex-direction: column;
		justify-content: center;
		align-items: center;
		width: 100%;
		height: 100%;
	}

	#modal {
		overflow-y: auto;
		overscroll-behavior: none;
		position: relative;
		background: white;
	}

	#controls {
		position: absolute;
		top: 0;
		right: 0;
		width: 3rem;
		height: 3rem;
		font-size: 2rem;
	}

	button {  /* reset */
		cursor: pointer;
		border: unset;
		margin: unset;
		padding: unset;
		background: unset;
		color: unset;
		font: unset;
	}
	button {
		position: fixed;
		width: inherit;
		height: inherit;
	}

</style>

Don't snooze on Lit either though. Again, I see any custom element library being essential for use with Alpine because it just gives Alpine more rich tags to use.

Here are two combos that would make me happy.

1. HTMX + Alpine.js + Lit
2. HTMX + Alpine.js + Ponys

or even offering 2 custom element libraries as options (they're both ridiculously tiny).

3. HTMX + Alpine.js + Lit + Ponys

NOTE: Lit offers a downloadable non-NPM single file option. The difference is that it doesn't use decorators and so the boilerplate per component is just slightly more annoying without any other differences.

<script src=" https://cdn.jsdelivr.net/npm/lit-html@3.1.2/lit-html.min.js "></script>

Reddit post about various HTML component libraries

[Simon-Martens]

A lot of this stuff is really cool, thanks for the update. +1 for the user defined sql build.

As for the frontend, I really like the idea of alpine.js; also, htmx would allow for creating html snippets for page layout, records, tables and settings on the server and sending it down to the client -- if someone wouldnt like one component it would be easily replaced by creating a template (not even having to code in go). But just choose something you feel comfortable working in. I really like the idea of being able to insert custom html components and all that stuff, sounds really great.

Also, merging the Dao with the App instance and allowing migrations to access all the additional data feels like a logical step.

As for the RecordUpsert thing, I personally think it's fine as it is and easily understandable.

Good luck with your sabbatical and the refactoring.

[Simon-Martens]

• i must say, i like the pocketbase website for documentation a lot: it might be one of the best documented projects i've seen. I like that it is concise, states the most important facts plain and short (so that you don´t have to read through a lot of text), and that it has a lot of example code.

[jonshipman]

I've done some work around svelte and compiling into web components. See a boilerplate I made here: https://github.com/jonshipman/web-component/tree/main/web-component

Depends on if you've already started working on it, if not, you could keep the admin ui in Svelte (less work on you) and just break the pieces up into web-components.

Setup an npm workspace and separate each piece into a separate svelte package. Then the packaged frontend could be web-components. e.g.

<pb-wrapper>
  <pb-menu></pb-menu>
  <pb-collections></pb-collections>
  ...
</pb-wrapper>

In another project, I wrote a reusable use:action listener https://github.com/jonshipman/jonshipman.com/blob/main/src/lib/util/listener.ts. It can be used instead of the event dispatcher to send native Custom Events that can bubble, so you can send deep events up to the top component if you need to.

And so on. Only pitching in if it makes your work easier.

[tigawanna]

can components be embedded post build?

[jonshipman]

Yes. To be clear, this would be using "Vite" and not "SvelteKit." You can use sveltekit, but I found it introduced more headaches (required a second build for example).

The first project I linked to is a boilerplate for WordPress plugins. Have a few custom plugins for clients out there and refactoring them into web-components using this setup. On the PHP side we're just transforming the [namespace-shortcode] to a <namespace-shortcode> and passing down the attributes into the component.

For this project it could be simpler as you don't need to split it into multiple files (he could do one project). For my use, I split it up so WordPress can enqueue the files when the shortcode is called.

One way I'd imagine it would be you could have a dictionary and declare things like "pb-wrapper" => "pb-wrapper". Then in an api doc, a user could override "pb-wrapper" with a custom web-component (written in whatever they want) and have it accept the same props that the built in pb-wrapper uses.

[tigawanna]

this definitely sounds like a win

[jonshipman]

Just thinking outloud, but the dictionary could be as simple as a pb_web folder and having files named "pb-wrapper.js" that would replace pb-wrapper. So if you wanted to overwrite a particular component, you just have to create a customElement that defines "pb-whatever".

pb_web/
├─ pb-wrapper.js

class MyPbWrapper extends HTMLElement {
  connectedCallback() {
    const shadow = this.attachShadow({ mode: "open" });
    shadow.innerHTML = '<slot></slot>';
  }
}

customElements.define("pb-wrapper", MyPbWrapper);

[exside]

riot.js is round for a long time, stable, powerful and I have used it in many projects, syntax is very close to svelte and components can be compiled in-browser if one wants to avoid a build step (I'm doing this all the time when developing, can help with the setup if interested).

[ganigeorgiev]

This post seems to have started some frontend frameworks war which wasn't my intention.

The Admin UI rewrite is the least important part of the announcement. As mentioned in the post and the other replies regarding this - nothing specific is set in stone yet and I'll provide more details once I start working on this part of the refactoring.

The Go APIs refactoring is with higher priority and has bigger impact for the PocketBase users as it will introduce breaking changes.

So please be considerate, read the previous replies and keep the comments on topic to avoid the unnecessary noise.

[Odas0R]

Not pointing out my web preferences, but instead I'm sharing my appreciation for your work, while fully onboard with all the changes <3

Thank you!

[uazapi]

Hello Gani,

I commend you for your initiative and transparency regarding the refactoring plans for PocketBase. The proposed changes are exciting and signal a promising future for the project.

I would like to suggest a special emphasis on features that could significantly elevate the efficiency and utility of PocketBase: support for bulk operations, such as bulk insert, update, and delete. Given SQLite's limitation of allowing only one write at a time, the ability to process transactions in bulk would represent a significant improvement. Currently, to circumvent this limitation, we often resort to loops for insert, update, or delete operations, which can overload and potentially lock up the database. Implementing bulk operations could reduce the load on the database, avoid locks, and, consequently, improve the overall performance of the application.

Furthermore, the functionality of importing and exporting data in the admin panel would simplify data management, making PocketBase an even more versatile and accessible tool for a variety of use scenarios.

I recognize the challenges and the current focus on refactoring, but I believe these features could bring considerable benefits to PocketBase, both in terms of performance and flexibility.

Thank you for your attention, and I look forward to following the project's updates. Success in implementing the planned changes!

Best regards,

[rajeshg]

Like where Pocketbase is heading.

Can we have better community involvement?

Once the initial prototyping is done and when you decided on a plan of action, if you could post issues with a help-needed tag, I am sure there are enough people with interest in this project. Having more community contributions is good for the overall long term health of this project. It comes with it headaches, but the tradeoff could be worth it.

[iamdineshkumar]

Hello @ganigeorgiev

I feel following can to be included

1. As mentioned by @uazapi , bulk insert, update and delete operations can be included. Same can be included in Admin panel as well since right now there is no direct option to do that.
2. Allow aggregation operations is another need in certain cases. I know right now view collection is doing their job well. Instead of that, if there is an option to handle queries with option to pass argument will be great
   For eg
   select * from articles where author_id = ? and category = ?
   In above case we have two arguments and we can declare similar to fields like userId string and category string. If we declare that during creation then we can validate arguments as well as their type before execution of query as well as we can validate rules also.
   This will reduce creating view collection to do aggregation operations as well as will increase performance since we are going to filter the result instead of creating huge data set using view collection and then querying it.
3. DB refactoring to use database/sql gives a window to support other SQL dialects which is a awesome news. Since many are trying to achieve that by customizing pocketbase. we need to refactor backup process of DB in case of other type of DB.

[VittorioParagallo]

I have a suggestion: the resulting json has the system fields id, created, updated on a different level then the other fields (data nested level). I'ld suggest to keep all the collection fields at the same level in the response. It's a big breaking change, so i guess it's worth of a deep thinking before v1.

[JunyanHex]

Sorry for the intrusion，is it possible to customize the default sorting for each collection？Rather than specifying sort at params, can it be predefined? This way, the JSON obtained through 'expand' (relation and back-relations) would also follow the predefined sorting.
I suppose this feature would be well-suited to be added in the “API rules” section, where it could determine the default sorting.

[parkuman]

+1 for the re-usable standalone component library; I love the design you've created with PocketBase!

I've been experimenting with building out a small CMS based on PB, using all the same components as you to keep a cohesive style. Rather than extending the UI, I've made the CMS its own admin dashboard entirely using PB as a framework (based on @ganigeorgiev 's response in #50). It is essentially a stripped back version of the PB admin dashboard, with limitations in place for what the users/clients can do. Having the re-usable components would help reduce a lot of the copy-pasting of component files and helpers that I've been doing. I will share a link soon once I think its usable.

Maybe with these upcoming refactors and better admin UI extensibility a CMS project would be better suited as an extension/plugin of the admin UI itself rather than a standlone dashboard? Would love to hear other people's use cases for extending the admin UI.

Edit: This would be perfectly suited to be an extension of the admin UI #3241