keystone v5

[htor]

@JedWatson
care to elaborate why the new keystone is being developed behind closed doors? as an oss project it's really essential that things are discussed out in the open, that the community can shape keystone into it's next form. if the community is left out (and invite-only members decide everything) the project will suffer. it does now – github issues+pr's are delayed infinitely because there's a secret v5 project nobody knows about. you even have a code name for it. why? when responding to bugs and pr's the devs say things like "please remind us later, we got lot's to do. will come back to you later". who are "we"? this is odd. open source is a collaborative effort and everyone contributing should be treated as equally important. at this point it's very obvious that keystone is being developed by and for the Thinkmill company, not the rest of the world.

[jesstelford]

Excellent question! I'm not sure why the decision was made to keep it private, but I do have a little more information that may help illuminate the situation a bit for you:

• The folks working on v5 (including me) are mostly not involved with v4 / the OSS packages that exist in the wild today
• This question has been raised internally at Thinkmill a number of times without resolution
  • However, there has been a specific intent to test v5 against real world apps to ensure we're not building into a corner, or repeating mistakes from <=v4
• v5 is being run pretty much as an OSS project within Thinkmill - multiple different projects/people using it, with different goals, contributing fixes and features as they come up.
• I want to open source it 😉

[Vultraz]

@jesstelford but will we (the general community) be able to use it when it's done? I really like Keystone, but there are a few really frustrating deficiencies with it (lack of nested object support in lists especially) that I was hoping would get resolved with further 4.x series patches. If 5.x is being developed in secret... will we ever see it? Or are we stuck with 4.x forever :(

[jesstelford]

will we (the general community) be able to use it when it's done?

100% yes! It's been built from the start with OSS in mind.

Ideally before it's "done".

Personally; I have a nascent business built on v5, which I am investing heavily into, so it's not going anywhere, and can only benefit from being OSS!

[jesstelford]

Re-reading your question, there's another interpretation:

will we be able to use it when it's done?

Like I said; it's being used in real world projects right from the start (from very small to very complex), so it's definitely usable.

Will it meet all expectations of an existing KS4 user? Probably not; it's a ground up rewrite, and we simply haven't gotten to all the usecases yet. But we (I) want it to support most usecases.

I'm not sure if you've poked around much how much you know, but a consistent theme in v5 is the amount of room for customising how it works, from different database adapters, to an extensive hooks API (not React hooks), to a detailed and well structured query schema. Hopefully this means that most usecases can be built on top of core without having to have things changed/added in v5.

[Vultraz]

a consistent theme in v5 is the amount of room for customising how it works

I actually haven't been able to find any info about v5 besides what's mentioned here and one small blurb on the Slack, so this is awesome to hear! I'm working on a project based on v4 right now and I'm banging my head against the wall due to the aforementioned lack of nested list objects, so hearing v5 will have more flexibility in general is great. 😃

Is there any general ETA for when it will become available? Or is that still very much up in the air? I'd kinda like to get my hands on it, since it might solve some of these problems I've been having with v4. 😛

[jesstelford]

I actually haven't been able to find any info about v5 besides what's mentioned here

D'oh, my bad! We haven't actually shared anything yet! 🤦‍♂️ (Because it really is run like an OSS project internally, I forget it's still not public)

Is there any general ETA for when it will become available?

This issue has prompted us (kicked us in the butt) to put together a proper timeline / plan, which we'll share shortly. But know that we've greatly accelerated the timeline to get it open sourced as soon as possible. Thank you! 🙏

it might solve some of these problems I've been having with v4. [..] lack of nested list objects

There is still no nested-list support 😅 The couple of times we've reached for that kind of functionality, we've realised that it's better expressed as a different data structure. Do you have a link to your specific usecase? While as yet we have no plans to build it into v5, it's good to have it in the back of our minds!

[Vultraz]

Do you have a link to your specific usecase?

@jesstelford I do, actually. I have a contract to revamp the entirety of this site. I'm rebuilding it from the bottom up since it's wayyy to old to simply be updated (it was built back in like 2001!). I decided to use Keystone as my CMS (I basically have total free reign over the design), and everything is mostly going fine, but I've run into a problem working on this page specifically.

My Keystone list looks like this:

... but my problem is Keystone only has a TextArray field. I really need a general object array field so I can create entries for the title subchapters without having to create hundreds of new documents in the database and link them with Types.Relationship.

In hindsight, perhaps I should have chosen a more mature, traditional Apache-based CMS (Drupal, maybe), but after exploring Node.js + Express I got hooked and went looking for a good Node-based CMS. I tried ApostropheCMS first, found its documentation lacking, and then settled on Keystone. It really is a good piece of software, but stuff like not having general-purpose Array support for database entries despite Mongoose and MongoDB supporting that out of the box is... frustrating. 😞

[jesstelford]

Oh yeah, we've hit that usecase ourselves; The need for a "write generic content" field.

Instead of going for a generic array-based approach, we've decided on a more specific type which handles that structure in a more formal way which will be part of v5. I don't want to over-promise, but it's way more powerful than nested lists / arrays ;)

[htor]

@jesstelford
thanks for your replies

Excellent question! I'm not sure why the decision was made to keep it private, but I do have a little more information that may help illuminate the situation a bit for you:

* The folks working on v5 (including me) are mostly not involved with v4 / the OSS packages that exist in the wild today

i feel the structure and processes inside Thinkmill shouldn't be something the community has to deal with. even if i were to try knowing what's going on inside there, i can't because you don't/can't share this information with the community.

I actually haven't been able to find any info about v5 besides what's mentioned here

D'oh, my bad! We haven't actually shared anything yet! 🤦‍♂️ (Because it really is run like an OSS project internally, I forget it's still not public)

this is suprising, given the size of this project. even if it's run like an "oss" project internally, it's not visible to the rest of the community! this resembles the cathedral model of developing, only your version more closed off and most of the individuals in the (informally) elected core group represent the a company and its interests.

will we (the general community) be able to use it when it's done?

100% yes! It's been built from the start with OSS in mind.

Ideally before it's "done".

Personally; I have a nascent business built on v5, which I am investing heavily into, so it's not going anywhere, and can only benefit from being OSS!

a lot of important decisions are made right now that later can't be reverted and there's going to be frustration around architecture, features and non-revertible constraints, because it's not discussed openly.

"keystone" as a project is open source and is the result of many peoples work and contributions. if you make a new version of it you should honor that work – not open source it when you think it's "done" and to get your bugs fixed.

[JedWatson]

Thanks for jumping in @jesstelford

@htor I'll try and give some background and answers to your concerns.

First off, I've said in several places (here and on Medium) that we (Thinkmill) have been exploring what the future of Keystone could look like if we had a "do over" combining all the learnings since I started the project over five years ago.

Not to re-iterate what I've said elsewhere too much but pushing forward with Keystone 4's architecture just wasn't working out on an effort to results basis, and for that reason we went back to a blank slate several times with Keystone 5. Including one nontrivial implementation we scrapped mid last year.

Everyone has different experiences, but I believe that sometimes creativity needs time, space and patience. Room to experiment without fear or judgement, or accountability. And importantly, without burning other people's time and energy.

For better or worse, that's what we chose to do with Keystone 5. I hope that, when the result is opened up, it will be worth it for everyone who's been involved in its creation, and who's been waiting for it. I also hope it's of value to people who've never heard of or tried the project before.

Keystone is by no means the only significant open source project to operate this way; recently, the React team dropped a massive new feature (hooks) that was a well-thought-out solution to a problem they knew well from both the community and their own use-case, and that was worked on either entirely or mostly behind closed doors until the announcement at React Conf last year. Because they took the time to get it right, it was universally well received.

Now, I'm personally glad that when we decided to scrap our previous take on "new keystone" it was behind closed doors. It was the right call, and we made it without fear of burning anyone outside Thinkmill. Nobody had based a project on it, nobody had wasted time contributing to it, and our next iteration (what will be the public version) was much better. Any project going through this level of reinvention is complex, and I've seen people get frustrated with the apparent lack of progress on Keystone, but I've seen other projects burn contributors and early adopters worse. Neither approach is perfect.

So that's what's been going on. The good news is we're basically done with all that and are ready to open it up - partly because we're confident and excited about what we've done, and partly because I think keeping it closed is really diminishing returns at this point. It was always meant to be shared.

open source is a collaborative effort and everyone contributing should be treated as equally important. at this point it's very obvious that keystone is being developed by and for the Thinkmill company, not the rest of the world

Lessons have been learned about the sustainability of open source projects, and if Keystone works for Thinkmill that's absolutely a good thing for anyone else who's using it or contributing to it. We got into problems maintaining version 4 because it was a beast, but stopped being the core of our business, and that's unsustainable. That left it to a few maintainers (including myself) who tried to continue on personal energy alone, which is a recipe for burnout and frustration. Keystone 5 is, among other things, a massive investment to change that so the project has a bright long-term future.

That doesn't mean Keystone is developed only for Thinkmill. We can freely give our work away, and benefit from the contributions of others, in a healthy ecosystem. These two things are not in conflict. All the documentation, guides, and support has been (in the past) and will be (in the future) done because fundamentally we love sharing what we build with others and collaborating with incredible people all around the world.

[Vultraz]

Honestly, @JedWatson 's reasoning makes a lot of sense. I myself am the project admin and lead developer for the open source game Battle for Wesnoth, and we've recently begun attempting a total ground-up rewrite in a new engine as our current handmade engine has proven inadequate for what we want to do. Design decisions made long in the past continue to make life hard for us today and has resulted in burnout and very little enthusiasm to continue work with the current codebase.

The Wesnoth 2.0 effort is being undertaken in a semi-private context (private Discord channel, limited repo access) since I felt it would be better and easier to keep the initial team small, and to avoid unhelpful comments from the peanut gallery and naysayers. It's a huge undertaking and having too many cooks doesn't help.

Given that Keystone 5 was also a total rewrite, it makes a lot of sense that they would develop it in private. As Jed said, it gives a lot more flexibility to experiment and toss out what doesn't work.

Really looking forward to seeing what Keystone 5 brings once you finally make it public. v4 isn't bad at all, and if you've managed to clear out a lot of cruft I'm sure v5 will be even better!

[jesstelford]

Including one nontrivial implementation we scrapped mid last year [..] I'm personally glad that when we decided to scrap our previous take on "new keystone" it was behind closed doors.

I want to reinforce this point with my own experience; I built a non-trivial project on that version of keystone (which was also a total ground-up rewrite). It got a lot of things right, and was really nice to work with. But there were still some fundamentally rough edges which meant either a lot of work poured into fixing it, or start from scratch (again) and also incorporate other learnings we gained from using that first rewrite. It also meant rebuilding the project. Thankfully Thinkmill was in a position to be able to spend that time/effort, and it's better off for it.

If we had open sourced it, then ditched it, then open sourced another one (all while KS4 is still alive and well), we felt it was not great for anyone.

Now with the upcoming v5, I'm really happy with the results so far, and we're at the point where we want to get everyone else's feedback (and help) - there are still features missing, and still room to shape the future on top of the core we've built behind closed doors, but I'm sure you'll really like what's been done so far!

I love the passion of the community - it's great to have you participating and raising these concerns @htor @Vultraz, thank you! <3

[Vultraz]

@jesstelford I'd like your opinion. Do you think it's worth putting my v4-based project on hold for awhile and wait for v5? From what you say it sounds like v5 will have a lot of things that would make my work easier, and I don't know if it's worth trying to clobber solutions together using v4. Figuring out a scaleable and maintainable (what I produce can't be fragile) solution for the lack of a generic content field especially is giving me massive headaches. The best thing I've come up with is to use the Code field and require manual editing of JSON data, but as you can imagine that is extremely apt to break.

Do you think it worth waiting, in your professional opinion? I'm really trying to weigh continuing with v4 or waiting a few (hopefully) months for v5.

[jesstelford]

I honestly can't say one way or the other. Because v5 is a ground-up rewrite, there are some very fundamental differences which may mean you have to port a lot of code over (including your client/rendering code), ultimately outweighing the benefits of the new field.

But you shouldn't have to wait months to see v5 - we've got some immediate plans which should help you get an answer sooner.

[Vultraz]

Oh, that's awesome! I'll definitely be keeping a lookout.