feat: db-plugin layer to support multiple dbs (sql, nosql, file, in-mem, etc)

[jensim]

I find this project totally lovable for open source/hobby projects that i deploy on Heroku. But the description of this project is that it is enterprise ready. Thus I expect to be able to choose my database type, ie Oracle. The project is however intwined with Postgres. But this could be cured, and we would be even more ready to claim enterprise approval.

The knex dependency, which is already in use for CRUD operations, but not for migrations, even though the knex website claims to have this capability with handling of differences in SQL syntax.

We might be able to break the Postgres dependency by dropping db-migrate and moving over to knex for db-migrations and connectivity. The pg-dependencies would then belong in dev-dependencies.

Acceptance criteria:

• Backwards compatibility for db-migrations
• Make the postgres dependencies into dev-dependencies
• Connection config adapted according to the knex json standard
• Connect to MySql or OracleDB by adding dev-dependencies like so http://knexjs.org/#Installation-node and adding config for connectivity.

[ivarconr]

Thanks for reaching out @jensim!

I agree that it could be easier for (some) enterprises to take advantage of unleash if the project supported a range of databases.

It would also be kind of nice to rely on knex for migrations and drop the db-migrate dependency.

My only feedback: A rewrite of the migrations must be done with backward compatibility in mind, so that we have an upgrade path for existing users.

Is this something you, @jensim, would like to work on?

[jensim]

I might be able to work on this in some spare / slack time - but I cannot make any promises. It would be fantastic though, and I will try to make time soon.

Good feedback, @ivarconr about the existing user base - i'll make that intent clearer in the acceptance criteria about the migrations compatibility. I'll have to make an investigation into how db-migrate and knex work in practice.

[ivarconr]

...and of course there is always a chance we have some postgres specific syntax somewhere, but that should be possible to tackle!

[jensim]

Hopefully the acceptance criteria of being able to run the application on other databases will catch any and all the postgres specifics. =)

[matthewt]

Hi there. I'm interested in this project but also would appreciate some database independence. Activity on this seems to have stalled is there a reason for that?

[jensim]

Im only stalled due to lack of time. I worked on this in my free time. There are some postress specific syntax scattered throughout the project as well as postgress dependencies nested into the core dependencies and run config and travis testing.
I'd prefer fixing everything, but I bit of more than I could chew of. Now Im a bit swamped. Please take up where i left of @matthewt 😊👍

[Lewiscowles1986]

There is a lot of work on this in the https://github.com/dustinbarnes/unleash/tree/divorce-from-postgres branch which I'm working on.

[Lewiscowles1986]

Current works fit into the following statements

• move the parts which should be storage backend independent to common shared folders
• rename things containing Db to be generic (unless they are specific)
• try to eek out an interface which is lean and to the point, attempt to ignore DSL
• make the db (uses knex) the default implementation

@dustinbarnes I'm not done yet, but check the branch I'm moving closer.

[Lewiscowles1986]

It might be an idea to see if travis matrix builds can be used to speed the various official supported database back-ends.

Most of my time has been fighting an overly terse test suite. I love minimalism and think the output is great in green runs, but it gives me few clues during red runs or partially failed runs.

[mpeyper]

Is there any update on this? Is there any way I can help move it forwards?

[smijran]

Same here - please also update us with status.

[jogaco]

@Lewiscowles1986 can we contribute somehow to your branch ?

[ivarconr]

So I do not want to support all databases as part of unleash-server code. What I do envision for a future version of unleash is to have a proper "store" abstraction. This would allow anyone to implement an adapter for any type of database (even in-memory) and use that together with Unleash.

This work require some rewiring of exiting code and a clear "store" contract. If someone can contribute on this work, that's awesome. My priorities are currently elsewhere for Unleash.

[jensim]

So I do not want to support all databases as part of unleash-server code. What I do envision for a future version of unleash is to have a proper "store" abstraction. This would allow anyone to implement an adapter for any type of database (even in-memory) and use that together with Unleash.

Sounds like a smart design choise! Then would migrations maybe be supplied like a flyway bundle beside the sql-store-plugin?

[jensim]

This work require some rewiring of exiting code and a clear "store" contract. If someone can contribute on this work, that's awesome.

Sad to say that I am also short on tim + otherwise occupied as well 😔

[mpeyper]

This work require some rewiring of exiting code and a clear "store" contract. If someone can contribute on this work, that's awesome. My priorities are currently elsewhere for Unleash.

I'd like to help out if I can. I've got the project running locally and have been familiarising myself with the code.

Did you have an interface in mind for the store abstraction because I'm struggling to find a clear one works with all the existing stores, at least, without embedding a lot of knowledge about the models used within the system.

[ivarconr]

Did you have an interface in mind for the store abstraction because I'm struggling to find a clear one works with all the existing stores, at least, without embedding a lot of knowledge about the models used within the system.

Actually, I was thinking that part of this work is also to better model the data used and have those as part of the contract for the stores to implement support for. An object store could store json blobs directly, while other stores would need to serialize the data a bit.

Some loose thoughts:

• I am not married to the current store implementations for postgres and think those can be improved a lot.
• I believe we will end up with a service layer between the controllers and the stores, doing some of the logic that is not just about storing and retrieving data.
• Typescript would be ideal for the contract at least, making it much easier
• Migrations should be a concern of the store plugin. Current migrations would move to the postgres specific implementation.
• This will open up a lot of different types of storages: in-memory, file-based, redis, NoSql, SQL*, etc.
• We have to consider having the stores emit events, this will be crucial in order to enable stream-based api for the clients.

I think the first step would be to start a small specification document for this effort and maybe focus on one or two store types.

[mpeyper]

Just for my own clarity, do you foresee the store plugins working in genric terms only (e.g. create, read, update, delete) or at the domain level (e.g. addFeature, getFeature, saveFeature removeFeature).

I see the generic form as being easier to maintain in the long term, but limited it how optimised each plugin could be to take advantage of the underlying technologies.

Option two would have a much larger API surface area and would require each store plugin to be update each time a new domain is introduced.

I prefer option 1 myself.

[Lewiscowles1986]

mpeyper, have a go at documenting the API and getting the tests to consistently pass, there is a PR with some work, but the problem was always how flaky tests were and how the project was organised. It's less of a part-time effort and more of a large undertaking

[mpeyper]

@Lewiscowles1986 things have become busy at work for me so I haven't had a chance to look at this at all again. Assume I'm not working on this unless I comment again.

[zivyatziv]

While we're at the subject, I'd like to suggest migrating from knex to sequelize.
https://www.npmtrends.com/knex-vs-sequelize
https://npmcompare.com/compare/knex,sequelize

What do you think?

[jensim]

While we're at the subject, I'd like to suggest migrating from knex to sequelize.
https://www.npmtrends.com/knex-vs-sequelize
https://npmcompare.com/compare/knex,sequelize

What do you think?

Does sequelize handle DDL statements? knex does. What other gains come from thar migration?

I started poking around in this 2 years ago, as I had need for toggles. The need diminished, and so did my efforts here, I'm afraid. 😔 Still contributing open source, but still in line with my personal needs at the time 😇 this has ended up quite far down my wait-list

[ianpittwood]

Hi, is this feature still in the works? My team is primarily a SQL Server shop so we'd love to see the expanded DB support. Thanks!

[ivarconr]

Hi @ianpittwood,

we have started to plan a technical overhaul of the entire code-base as part of v4. In short we plan to:

• migrate to typescript
• add a "service layer" between stores and controllers
• simplify and standardize all stores and provide a official interface for the stores
• make it possible to plugin-in your own store implementations (which will be responsible for storing and retrieving).

In short we thus will make it pretty easy to integrate with "any store provider", even a S3 bucket. This will make unleash much more flexible.

We have scheduled to start the actual coding of this in beginning of 2021 as this will require some full time focus (and we have some new full-time resources coming on-board at this time)!

Is this something you would like to contribute on?

[chaeron]

Related issues were closed, but I don't see any way to route Unleashed persistence to a different database (MongoDB in my case).

I would love to use Unleashed for a very high profile healthcare project, but unless there is a framework for us to be able to at least write our own storage adapter for MongoDB, I can't justify it.

So....what's the status on supporting something other than PostgreSQL for persistence in Unleashed? It's not clear from the series of closed issues I followed and how a Service layer might impact that.

Creating a plug-n-play persistence layer with a generic API shouldn't be that hard a task....

Thanks!

[Lewiscowles1986]

The status is write the code and submit a PR.

[chaeron]

In which case, we'll just write our own feature management solution, which will take less time and effort. Problem solved!

[ivarconr]

Hi, we do plan to provide the abstraction to allow users to provide their own persistence implementation. This is a bit in the future for our road-map and we are full focus on Unleash v4 which will form the technical foundation we need to create a standardized store layer.

(fyi: there do exists a old fork adding mongo. Not sure if it is maintained or even works: https://www.npmjs.com/package/unleash-server-mongo)

[Lewiscowles1986]

This branch is 1128 commits ahead, 1758 commits behind Unleash:master.

Look, solving by direct binding to something else. That is going to be a problem.

Problems I had were that the existing test suite wouldn't consistently pass from the start, and the design was a little clunky. I was under a pressure to pick a feature management solution, and a few spikes with unleash showed it was a bit of work. Nonetheless if your approach is to write your own... I don't think you'll have that good a time.

[ivarconr]

We have no immediate plans to support more store technologies.

[chaeron]

So...that means we'll not be using your software for feature management.

[thedanchez]

We have no immediate plans to support more store technologies.

This is a real unfortunate shame. It truly is. This is something that will literally block enterprises from considering this tech as it's not flexible enough in this key area.

[Tymek]

We have no immediate plans to support more store technologies.

This is a real unfortunate shame. It truly is. This is something that will literally block enterprises from considering this tech as it's not flexible enough in this key area.

For everybody working for an enterprise: let's to talk about what is blocking you from adopting Unleash and how we can solve it.

This is a reoccurring question. As far as I know we never had this raised as a blocker. Please help us find a compelling real-world use cases for other databases. You can voice your concerns here, or when talking to sales, but also to customer success team and developers from product team on our community Slack.

If setup is an issue, with our Pro offer we can take care of DB provisioning and all the related backup and maintenance.