Discussion: Development experience for third party apps

[marcelklehr]

Hello,

I'd like to talk about something that has me frustrated for a while. The nextcloud ecosystem is dependent on third party developers integrating new fabulous applications into the platform and some people actually want to do that. However, at least for me personally, the development experience leaves a lot to be desired. The documentation is the bare minimum and sometimes even outdated while large chunks of new developments in core are simply not accessible to third party developers when just looking at the documentation. People are expected, it seems, to dig through the API docs and make sense of the interfaces on their own. This is a problem in my opinion.

I feel like at the very least there needs to be:

• a thorough refactoring of the app dev manual so that it covers not only the PHP Controller infrastructure, but also caching, system tags, collections, client-side APIs, the vue ecosystem and possibly more that I haven't heard of yet
• some kind of newsletter/mailing list/forum where new developments in core are announced to developers (preferably ahead of time) in order to adopt and integrate them
• a place (online) for developers to share expertise and discuss problems

[Spartachetto]

Even if I am not a developer, I had the same impression trying to help with the issue #13330 .

It seems that integrating system tags into an app is not superintuitive.

[ChristophWurst]

Thanks so much for bringing this up!

This is something that also bothers me because I see that it doesn't only affect third party devs but also core developers sometimes.

We definitely should try to be better with documentation. The dev docs should cover all the public APIs and show examples of how to use them. It's easier said than done, though, so any help is appreciated as we don't have a dedicated person keeping the docs up to date.

For changes I like the format of Laravel: https://laravel.com/docs/5.6/upgrade. It's basically a prose version of the "Critical changes for devs" issues with a bit more guidance on how to migrate away from deprecated APIs or how to adapt to new ones.

For the js APIs I opened #15932, where we now have lots of new js packages that I hope are more straight forward to use than the hard to discover global variables/functions used in the past. I'm trying to document the usage of these packages as good as possible. Here is an example: https://nextcloud.github.io/nextcloud-event-bus/. Of course this also kinda falls short as nothing links to it. So as long as that stays undiscovered, it doesn't matter how good the docs are. I have a task on my todo list to add a page about js APIs to the dev manual linking to the specific packages.

[ChristophWurst]

a place (online) for developers to share expertise and discuss problems

That would be https://help.nextcloud.com/c/dev

[tobiasKaminsky]

as we don't have a dedicated person keeping the docs up to date.

Those who create/add/modify API, should then also adjust it in documentation.
Not sure how we can enforce this via some tools?

[noci2012]

And a valid & working demo of how to use it in stead of a crammed together hotch potch half baked non working example that could never have worked in the documentation. The provided example should also get ported then dev's can compare changes if needed there as well.

[Spartachetto]

@ChristophWurst here is a discussion on this topic in that section of help.nextcloud: https://help.nextcloud.com/t/state-of-app-development/63670

As you can see similar problems are raised and nobody is answering...
I understand the added workload, no idea if Nextcloud could use some automation to deal with this problem.

[noci2012]

@ChristophWurst: in addition:

We definitely should try to be better with documentation. The dev docs should cover all the public APIs and show examples of how to use them. It's easier said than done, though, so any help is appreciated as we don't have a dedicated person keeping the docs up to date.

The current documentation is incomprehensible and in consistent. Asking "new" developers to update it will probably not help there is more needed than "just" a few changes / clarifications.
The current documentation resembles a list of methods & parameters. I guess a howto / whento / whydo type of manual is needed. Something a programmer can start on page 1 and work through to get a working plugin.

[ChristophWurst]

Yes, makes sense, @noci2012. I'll try to write some docs the next few days and weeks.

Could everyone please name their most important pain points? Then I can prioritize what's most needed. I've worked with Nextcloud too long to be able to tell what is simple to understand or obvious and where we have to turn magic into something understandable. ✌️

[ChristophWurst]

Docs on the new JS APIs: nextcloud/documentation#1779
Docs on new events API: nextcloud/documentation#1780
First shot at an upgrade guide: nextcloud/documentation#1781

Just first versions, will iterate and improve later. Feedback welcome 🙏

[ChristophWurst]

a thorough refactoring of the app dev manual so that it covers not only the PHP Controller infrastructure, but also caching, system tags, collections, client-side APIs, the vue ecosystem and possibly more that I haven't heard of yet

Guess we should also find a better structure in the dev manual. Currently it's just this list of a few documented APIs. Maybe a split into general, front and back-end makes sense.

[ChristophWurst]

Those who create/add/modify API, should then also adjust it in documentation.
Not sure how we can enforce this via some tools?

So far the theory. But as we know not everyone likes writing docs. But this is fine. IMO it would be already helping if the PRs that need doc (updates) get a special label, like needs-docs or whatever. And then someone who doesn't mind writing docs can pick that up in a batch.

[marcelklehr]

Thanks for taking this on!

Basic things lacking documentation that come to my mind are access to user files, setting up background jobs, adding entries to the activity stream, as well as collections and I think admin settings. User and session management are rather easy to figure out from the API docs, but might benefit from some love as well.

[wiswedel]

IMO it would be already helping if the PRs that need doc (updates) get a special label, like needs-docs or whatever. And then someone who doesn't mind writing docs can pick that up in a batch.

That's even already there (however rarely used):

[ChristophWurst]

@marcelklehr got it. Will write something on these topics.

@wiswedel right! Let's use it more often again :)

[ChristophWurst]

setting up background jobs

Covered already https://docs.nextcloud.com/server/stable/developer_manual/app/backgroundjobs.html

[noci2012]

searching in the docs gives a lot of false positives.
f.e. app.php or "app.php" as a search string gives ALL references pages where app is (may be part of a word like Apps, Appl, Application etc. ) and php containing pages and both.....
Maybe some config on the webinterface can be done to at least make "" enclosed searches literal?

[ChristophWurst]

I think this is an issue of the theme. I'm not entirely sure but we use something like https://github.com/readthedocs/sphinx_rtd_theme.

[e-alfred]

Some parts of the documentation still refer to the Ownnote app which is pretty much unmaintained nowadays and never even got a release into the new Nextcloud App Store.

[juliushaertl]

Some parts of the documentation still refer to the Ownnote app which is pretty much unmaintained nowadays and never even got a release into the new Nextcloud App Store.

I did a bit of a rewrite of the app tutorial some time ago and it should be up to date with the https://github.com/nextcloud/app-tutorial repo. I also cannot find any reference of ownnotes in the docs.

This is by intention no application that would be in the app store as it is quite limited in its functionality to make it somehow easy to explain and also understand from a beginners developer perspective what the essential parts are. If you have any reference to outdated documentation, feel free to open an issue for that so we can have a look.

[korelstar]

I second the request for a better documentation of using tags in PHP. This may require some improvements in the API itself (see #19404).

[ChristophWurst]

For changes I like the format of Laravel: https://laravel.com/docs/5.6/upgrade. It's basically a prose version of the "Critical changes for devs" issues with a bit more guidance on how to migrate away from deprecated APIs or how to adapt to new ones.

This page is live now for latest and will become available with 19. See my topic at https://help.nextcloud.com/t/an-upgrade-guide-for-app-developers/77781 for details. I'll extract and backport the 18 changes now as well. I'd appreciate feedback on the format and usefulness so I'm not wasting time one something nobody would use :)

[marcelklehr]

Thank you for the changes that have already happened. The docs have seen major improvements!! 💙

The page that documents nextcloud Flow links to a (possibly outdated) slideshow that mostly contains uncommented code. It's a start, but it leaves me having to do a lot of digging myself on how to use these APIs properly, if I want to provide flow events and/or operations. 😿

[e-alfred]

Do you think occ app:check-code could be extended to also check for Javascript deprecations?

On a more general note, a lot of the developers of apps do it in their free time. Do you think that the upgrade experience could be made less time consuming so that a new Nextcloud release keeps apps working without major breaks that is problematic for users and app developers alike?

[ChristophWurst]

Do you think occ app:check-code could be extended to also check for Javascript deprecations?

We have https://github.com/nextcloud/eslint-plugin for that. See https://docs.nextcloud.com/server/latest/developer_manual/app/coding-style.html.

Do you think that the upgrade experience could be made less time consuming so that a new Nextcloud release keeps apps working without major breaks that is problematic for users and app developers alike?

That is impossible. This would mean that Nextcloud is never allowed to change any of its APIs. Having a decent deprecation guideline, announcing the critical changes and offering a detailed upgrade guide is the best we can do IMHO.

[blizzz]

The page that documents nextcloud Flow links to a (possibly outdated) slideshow that mostly contains uncommented code. It's a start, but it leaves me having to do a lot of digging myself on how to use these APIs properly, if I want to provide flow events and/or operations. crying_cat_face

https://docs.nextcloud.com/server/latest/developer_manual/app/flow.html ?

Yes, indeed that needs luv. The comments are in the Talk, but that obviously is not enough. Let me try to improve and extend it with the 20 cycle. Since recently we also have a dev Talk room for flow.

[Spartachetto]

To all the wonderful documentation - writing developers: according to nextcloud/bookmarks#1492 , integration with projects would need more documentation.

Hope this is the right place to raise the problem...

[ml31415]

From an admin point of view I'd like to add:

• Too many breaking changes / major version updates
• Frustration for plugin developers to constantly touch code, which was previously working fine
• As a consequence regular issues with plugins not being updated in time
• Plugins marked as out of date, even if they'd work perfectly fine without change
• Impossible to ship recent nextcloud version, if there are some plugins installed

And that's not only for third party plugins, but also ones featured by nextcloud. See here for example:
nextcloud/documentserver_community#298

[ChristophWurst]

Too many breaking changes

Do you have an example of a recent breaking change that affected you?

[ml31415]

No, I don't develop plugins myself. But if there were no breaking changes, then there shouldn't be a major version update, that forces plugin developers to update their plugins. Even if it's only to reevaluate compatibility, as seen with document_server, even that small update is something that is a constant maintenance burden, that not everyone is willing or has the time to do.

[ChristophWurst]

That is a fair point. The release schedule is documented at https://docs.nextcloud.com/server/latest/admin_manual/release_schedule.html#major-releases.

The release cadence is also discussed in https://help.nextcloud.com/t/major-release-cadence/161685.

We could probably decrease the number of major releases and have a few minors in between. Then the breaking releases occur less often.

[aaronhuggins]

@ChristophWurst What might be really nice is for some way for developers to "declare" that an app version is compatible with a new released Nextcloud server version without having to re-publish the app. Is such a thing already possible? It looks like we would have to do a patch version of nextcloud/documentserver_community in order to advertise to Nextcloud server that it is compatible. I can't think of a way that's documented by which we could accomplish that. Maybe such a thing is a separate discussion?

I'm a willing contributor to documentserver_community without access and it looks like that contributed to opening this issue here.

[ChristophWurst]

You'd have to put that meta data somewhere. Isn't that as much maintenance burden as changing info.xml?

[ml31415]

Any maintenance burden, that can be avoided or automated, should be avoided. All of us, especially and including plugin developers have enough other work and projects around, that any unnecessary burden leads to frustration and more unmaintained plugins.

[marcelklehr]

We could probably decrease the number of major releases and have a few minors in between. Then the breaking releases occur less often.

I think this would greatly improve quality of life for developers!

[ChristophWurst]

The upcoming iteration will only have a minor bump of server with no breaking changes. Only the next release will be a major again due to some breaking changes of the PHP platform we have to adapt to.

Ref https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule/17316db0a7d1800a57f513e00cdf7622f0b4fa1a#nextcloud-271

Apps that worked with 27.0 will work with 27.1 without modification, unless their info.xml specifies "27.0" as maximum Nextcloud requirement. I have never seen that in the wild.