[UX] Create a project browser for modules, themes, layouts etc

[docwilmot]

See backdrop-ops/backdropcms.org#56 (comment)

So that users can simply select projects to download within update module, rather than the not-so-friendly cutting and pasting of zip file urls.

I would personally like to suggest the ability to select multiple projects for download in one go, and related, automatically attach dependencies.

Any ideas?

[quicksketch]

There's actually a really great Google SOC project that built out this functionality years ago in Drupal 7: https://www.drupal.org/project/project_browser

We could check out the feasibility of that project as a candidate for core. I'm also in support of this idea.

[cellear]

The story told in this thread about Wildkatana's attempt to get Project Browser functionality into Drupal is a fascinating look into the Drupal infrastructure. I'm amazed and very impressed by his patience and perseverance.

https://www.drupal.org/node/1243332

It strikes me that most of the objections raised in this thread don't really apply to us. Perhaps we should give some encouragement to Wildkatana to implement it here? It sounds like most of the work that we would need to use this for Backdrop is already done.

[quicksketch]

Thanks @cellear, that thread is indeed very interesting. It looks like the project browser functionality has pretty much been "ready" for a long time, with the main holdup being getting access to a complete development environment. 😦

And yes, I think the main hold-ups don't apply to us. For our initial pass, we could have the search box use the built-in search functionality instead of Solr (although we could set that up as well).

[Drupaldise]

This would be great!

[docwilmot]

Very cool. Would be fantastic addition.
Have left a note on the issue queue for the contrib project on D.org.

[Drupaldise]

I am interested in working on that... not sure if we need the old Drupal version.

[Bojhan]

I actually did the design and mentoring of this project for core. For some reason we never got there. The challenge isn't really the d.o part, although in the Drupal world it is.

Outside the Drupal world the challenge should be, a nice experience - which is kinda tricky with uploading and enabling. I find the Wordpress browse experience quite nice, but then installing it feels "Drupally".

[Drupaldise]

Sooo... how should we go further with that?

[quicksketch]

@Drupaldise: If you're up for it, we can get you access to the backdropcms.org server so you can make a database dump and set up a local installation of the site with the content types for modules, themes, etc. all set up; then work on porting the Project Browser Server module. Then you can set up a second installation of Backdrop to point at your local server, and start porting the Project Browser module.

Really you don't need an installation of BackdropCMS.org, for starters you could just install https://github.com/backdrop-contrib/project on the local installation that will act as the server.

[Drupaldise]

Ok, but I am not sure if I can handle this alone... maybe there is someone else there, who would be interested in working on that.

[Graham-72]

@Drupaldise It looks a very interesting and worthwhile project. I would be interested in working on it.

[docwilmot]

In my excitement I had started porting the client module. I've got the initial backdrop work here:
https://github.com/docwilmot/project_browser
Please fork at will.

Did this 2 weeks ago, and never checked to see if it works, so just a start, will need some love I suspect.

[docwilmot]

And start for project_browser_server too.
https://github.com/docwilmot/project_browser_server

[docwilmot]

This is the return that project_browser expects from the server (page 1); Dump from a Drupal install. We could hardcode that into project_browser.inc as $results_raw (line 486) to get the client module to at least display something useful while testing.

Cant help more than this for now.

array(
  'total' => 10594,
  'projects' => array(
    'better_exposed_filters' => array(
      'type' => 'module',
      'title' => 'Better Exposed Filters',
      'name' => 'better_exposed_filters',
      'drupal version' => '7',
      'author' => 'mikeker',
      'description' => 'The Better Exposed Filters module replaces the Views\' default single- or multi-select boxes with radio buttons or checkboxes, respectively. Description fields and Select All/None links can be added to exposed filters to make for a better user experience....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => 'https://www.drupal.org/files/styles/grid-3/public/images/BEF_0.PNG?itok=xd8JKJ2n',
      'usage' => '77849',
      'project url' => 'https://www.drupal.org/project/better_exposed_filters',
      'project status url' => 'http://updates.drupal.org/release-history/better_exposed_filters/7.x',
      'last updated' => '1428077881',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'smtp' => array(
      'type' => 'module',
      'title' => 'SMTP Authentication Support',
      'name' => 'smtp',
      'drupal version' => '7',
      'author' => 'José San Martin',
      'description' => 'This module allows Drupal to bypass the PHP mail() function and send email directly to an SMTP server. The module supports SMTP authentication and can even connect to servers using SSL if supported by PHP....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => '',
      'usage' => '75620',
      'project url' => 'https://www.drupal.org/project/smtp',
      'project status url' => 'http://updates.drupal.org/release-history/smtp/7.x',
      'last updated' => '1420662780',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'admin_views' => array(
      'type' => 'module',
      'title' => 'Administration Views',
      'name' => 'admin_views',
      'drupal version' => '7',
      'author' => 'damiankloip',
      'description' => 'Replaces administrative overview/listing pages with actual views for superior usability....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => '',
      'usage' => '74042',
      'project url' => 'https://www.drupal.org/project/admin_views',
      'project status url' => 'http://updates.drupal.org/release-history/admin_views/7.x',
      'last updated' => '1424423581',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'content_access' => array(
      'type' => 'module',
      'title' => 'Content Access',
      'name' => 'content_access',
      'drupal version' => '7',
      'author' => 'fago',
      'description' => 'This module allows you to manage permissions for content types by role and author. It allows you to specifiy custom view, edit and delete permissions for each content type. Optionally you can enable per content access settings, so you can customize the access for each content node...',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => 'https://www.drupal.org/files/styles/grid-3/public/images/content_access.png?itok=JG40rhvF',
      'usage' => '73543',
      'project url' => 'https://www.drupal.org/project/content_access',
      'project status url' => 'http://updates.drupal.org/release-history/content_access/7.x',
      'last updated' => '1366014322',
      'maintenance status' => 'Seeking co-maintainer(s)',
      'development status' => 'Maintenance fixes only',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'views_php' => array(
      'type' => 'module',
      'title' => 'Views PHP',
      'name' => 'views_php',
      'drupal version' => '7',
      'author' => 'casey',
      'description' => 'This module allows the admin account (user 1) to add fields, filters and sorts to views which use PHP code. It also supports additional accounts when the core PHP Filter module is enabled.',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => 'https://www.drupal.org/files/styles/grid-3/public/project-images/IMG_20140930_162850.jpg?itok=8mDJVuxU',
      'usage' => '71181',
      'project url' => 'https://www.drupal.org/project/views_php',
      'project status url' => 'http://updates.drupal.org/release-history/views_php/7.x',
      'last updated' => '1390577312',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'votingapi' => array(
      'type' => 'module',
      'title' => 'Voting API',
      'name' => 'votingapi',
      'drupal version' => '7',
      'author' => 'eaton',
      'description' => 'VotingAPI helps developers who want to use a standardized API and schema for storing, retrieving, and tabulating votes for Drupal content....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => '',
      'usage' => '71026',
      'project url' => 'https://www.drupal.org/project/votingapi',
      'project status url' => 'http://updates.drupal.org/release-history/votingapi/7.x',
      'last updated' => '1407995928',
      'maintenance status' => 'Seeking co-maintainer(s)',
      'development status' => 'No further development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'auto_nodetitle' => array(
      'type' => 'module',
      'title' => 'Automatic Nodetitles',
      'name' => 'auto_nodetitle',
      'drupal version' => '7',
      'author' => 'fago',
      'description' => '"auto_nodetitle" is a small and efficient module that allows hiding of the content title field in the form. To prevent empty content title fields one can configure it to generate the title by a given pattern....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => '',
      'usage' => '71019',
      'project url' => 'https://www.drupal.org/project/auto_nodetitle',
      'project status url' => 'http://updates.drupal.org/release-history/auto_nodetitle/7.x',
      'last updated' => '1307449915',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Maintenance fixes only',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'simplenews' => array(
      'type' => 'module',
      'title' => 'Simplenews',
      'name' => 'simplenews',
      'drupal version' => '7',
      'author' => 'miro_dietiker',
      'description' => 'Simplenews publishes and sends newsletters to lists of subscribers. Both anonymous and authenticated users can opt-in to different mailing lists. HTML email can be sent by adding Mime Mail module for D7 and Swiftmailer for D8....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => 'https://www.drupal.org/files/styles/grid-3/public/images/Subscriber_Management.png?itok=GnWqxzq8',
      'usage' => '62797',
      'project url' => 'https://www.drupal.org/project/simplenews',
      'project status url' => 'http://updates.drupal.org/release-history/simplenews/7.x',
      'last updated' => '1381947969',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'fontyourface' => array(
      'type' => 'module',
      'title' => '@font-your-face',
      'name' => 'fontyourface',
      'drupal version' => '7',
      'author' => 'sreynen',
      'description' => '@font-your-face provides an administrative interface for browsing and applying web fonts (using CSS @font-face, supported in all popular browsers) from a variety of sources. Try it out on simplytest.me....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => 'https://www.drupal.org/files/styles/grid-3/public/images/fontyourface_small_3.png?itok=iMNJHd7o',
      'usage' => '62197',
      'project url' => 'https://www.drupal.org/project/fontyourface',
      'project status url' => 'http://updates.drupal.org/release-history/fontyourface/7.x',
      'last updated' => '1367195415',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
    'uuid' => array(
      'type' => 'module',
      'title' => 'Universally Unique IDentifier',
      'name' => 'uuid',
      'drupal version' => '7',
      'author' => 'recidive',
      'description' => 'This module provides an API for adding universally unique identifiers (UUID) to Drupal objects, most notably entities....',
      'categories' => array(
        0 => 'admin',
        1 => 'search',
      ) ,
      'image' => '',
      'usage' => '61587',
      'project url' => 'https://www.drupal.org/project/uuid',
      'project status url' => 'http://updates.drupal.org/release-history/uuid/7.x',
      'last updated' => '1411454808',
      'maintenance status' => 'Actively maintained',
      'development status' => 'Under active development',
      'rating' => '0',
      'includes' => array() ,
    ) ,
  ) ,
);

[Graham-72]

@docwilmot Thanks very much for these - will you have time to answer any questions I may have once I study them?

[docwilmot]

Hi @Graham-72, I smile to see you ask that. I am no quicksketch sir. I'm pretty sure @Drupaldise and yourself are just as capable and very likely more so than I am. I was just trying to help, and give a start.
But if you ask, and I know the answer, will be glad to help. 😄

[quicksketch]

Thanks @docwilmot! Anyone who's interested in getting this rolling, I think it'd be a great idea to move these repositories into the backdrop-contrib group so that all contrib authors can collaborate together on it. I believe all 5 of us (so far) are already part of the contrib authors group, so moving it into contrib would let everyone commit on it. To follow procedure, it'd be great to list those interested here as maintainers in the README.md.

[docwilmot]

Transferred to backdrop-contrib

[Graham-72]

I have just modified project_browser.info and installed the module on a test site and it runs, giving me an excellent listing of Drupal modules and themes. Thanks @docwilmot for porting. I will now try to arrange for a listing of Backdrop modules and themes.

[docwilmot]

Cool.

I will now try to arrange for a listing of Backdrop modules and themes.

What's the plan? How you going to get that done?

[Graham-72]

What's the plan? How you going to get that done?

I see that project_browser_server includes an example with some static data for a few Drupal modules, so I am thinking that it would be best to get both modules fully ported, then get some static data for a few Backdrop modules pasted in manually, for test and demo purposes.

After that, we have to find out how we get the live data for backdropcms.org (I presume) once the lists are available there.

Does that sound sensible?

[Graham-72]

Question ... in project_browser.admin.inc line 26 he has

Add new repositories to use for the Project Browser, one per line, in the 'url|method|Site Name' format. MAINSITE_URL is added by default, and doesn't need to be set here.

What, if anything, should we use in place of MAINSITE_URL ?
Otherwise, I think porting is now complete.

[docwilmot]

What, if anything, should we use in place of MAINSITE_URL ?

I think we can ignore that for now; doubt we'll be using any other servers than the default once its up.

I am thinking that it would be best to get both modules fully ported

I am thinking though that the server just sends a list of available projects, but the project_browser actually has to do the fetching and downloading. I'd say the important task is to get the browser actually fetching projects from GitHub. And installing them. Once that's done we can get the server up to feed the browser live info.

[docwilmot]

Another small thing: I think the changes youve made so far directly to the project browser are reasonably obviously necessary, so this doesnt apply:

But I think when we start actually trying to make things work and write new code, that it may be best to to the changes in branches in a fork on your GitHub acct and send as PR to the Backdrop contrib repos. Sort of like how patches are proposed on D.org. You can merge them yourself even, but at least the rest of us get a chance to see what youre thinking.

What do you think?

[Graham-72]

@docwilmot

I am thinking though that the server just sends a list of available projects, but the project_browser actually has to do the fetching and downloading. I'd say the important task is to get the browser actually fetching projects from GitHub. And installing them. Once that's done we can get the server up to feed the browser live info.

Yes, maybe. Though I am partly thinking that backdropcms.org will be collecting this information from GitHub and so project_browser_server might be replaced by some equivalent module on backdropcms.org if all the data we need is already there. We need to find this out.

it may be best to to the changes in branches in a fork on your GitHub acct and send as PR to the Backdrop contrib repos. Sort of like how patches are proposed on D.org. You can merge them yourself even, but at least the rest of us get a chance to see what youre thinking.

Yes indeed. I was being lazy with this basic 'standard' porting stuff and will now start to use PRs for anything more.

[docwilmot]

I am partly thinking that backdropcms.org will be collecting this information from GitHub and so project_browser_server might be replaced by some equivalent module on backdropcms.org if all the data we need is already there.

The way I understand it, project_browser_server will be installed on backdropcms.org and will do the job of querying data from project module there and sending it to project_browser on client sites. Thats what @wildkatana has been trying to accomplish on D.org for the last couple years. On D.org the same would likely apply as here: project_browser_server would query project module, which in turn gets its data from D.org's Git repo where all projects are kept. Except we use GitHub.

In the end though, project_browser would get the info on available modules from project_browser_server, thats it. project_browser still has to go directly to the repo/download link to fetch the actual files. If someone set up a site anywhere, and project_browser sent a request to that site, and it returned that array I posted above, project_browser would work because the array tells it where the files are to download. The fancy 'server' part just makes it dynamic, and adds paging, and filters etc. We dont need that yet. We need project_browser to know what to do with the array it receives.

[wilderuncat]

If y'all have any questions, feel free to hit me up. Project Browser Server doesn't assume anything about the backend, it is just an API, so it is pretty extensible.

Regarding the multiple servers line, that was so that people could potentially host their own module repos/marketplaces down the line. Think Eclipse plugin repos, for example.

[docwilmot]

Seems like may be an issue with your PHP settings not supporting .zip extension, according to the Google. See this article.
Have you been successful installing modules through the standard update.module interface (admin/modules/install)? Try typing "https://github.com/backdrop-contrib/hopscotch/releases/download/1.x-1.0/hopscotch.zip" in there directly. If its a problem with your local setup ad .zip, you shouldnt be able to install there either.

[docwilmot]

Easier yet, see if zip is in the list of allowed extensions:

[docwilmot]

OK, seems we've proven that this works. I think any further discussion should move to the respective project issue queues?

I've made a change to the https://github.com/docwilmot/backdroporg_pbs module to use entityFieldQuery instead, to make it easier to add filters for project type and version (I added those fields to the Project nodes) so the project_browser theme and module tabs work (backdropcms.org isnt returning release data for any themes, so I added type = theme to hopscotch module to fake it). This is temporary until we can get a dump of BackdropCMS.org's database (waiting on @quicksketch) and/or longer term, when we can get the server on to BackdropCMS.org itself.

So now, we can focus on making project_browser sing and dance. I'm really excited about this, or as @jenlampton would say, I'm superexcited ©️ about this. Will be so great to see this in core eventually.

[quicksketch]

Awesome @docwilmot! Do you think this would hurt to throw it on BackdropCMS.org right away? Both the PBS and the accompanying module are pretty simple. What do you think?

One minor thing we'll change before putting it up: on BackdropCMS.org, all our custom modules that would normally be "backdropcmsorg_" are abbreviated to "borg_". 😉

[docwilmot]

all our custom modules that would normally be "backdropcmsorg_" are abbreviated to "borg_"

Believe it or not "borg_" is exactly what I prefixed that module, then I changed it, because who on earth would do that? 😄 I'll rename and push.

Do you think this would hurt to throw it on BackdropCMS.org right away?

There are a couple of things we havent quite worked out yet though. As noted in the borg_project_browser_server code, lots of things still arent mapped to project node fields because I dont know the field names. If you can find the time to plug those in, then by all means, having it on B.org would be great. It is quite simple after all.

Should I move borg_project_browser_server to contrib, seeing as it isnt really a module for public consumption?

[quicksketch]

Should I move borg_project_browser_server to contrib, seeing as it isnt really a module for public consumption?

Leave it out for now and we'll eventually merge it into the https://github.com/backdrop-ops/backdropcms.org repository with the other custom modules. When you think it's ready, just file a PR to add it like you would any other repository.

As noted in the borg_project_browser_server code, lots of things still arent mapped to project node fields because I dont know the field names.

For the most part, we de-fieldized a lot of project module so that properties would be consistent. However, I wonder if we're going to have introduced an issue because EFQ can't query other database tables besides fields.

to make it easier to add filters for project type and version (I added those fields to the Project nodes)

To clarify the setup on BackdropCMS.org, each project type is a different node type because we may need different fields on different types of projects (e.g. different taxonomy vocabularies). The "version" string is stored in the project_release_supported_versions table, as keeping track of the most recent recommended version gets a little complicated when dealing with multiple "API versions" (e.g. Backdrop 1.x.x and Backdrop 2.x.x).

[docwilmot]

For each project the server will need to get the following from Project:

project type
backdrop version
author
description
categories
image
usage
maintenance status
development status
rating
latest_release

Its going to be difficult to code/test this without knowing how the database/fields save data. I suspect a database dump may be the best idea if permitted.

[quicksketch]

I suspect a database dump may be the best idea if permitted.

I'll contact you via e-mail and we can get this arranged.

[docwilmot]

During the last Hangout, @quicksketch asked to mention status of PBS and PB. AFAIK, PBS was still serving up faulty paging (see backdrop-ops/backdropcms.org#73); I cant fix it or test it because thats probably an issue with Backdrop.org's implementation of Project module.

There was also a faulty PR which I pushed (sorry again) at backdrop-ops/backdropcms.org#103, which @quicksketch reverted, but I fixed it and pushed again; that needs looking into as well.

I want to start working on PB itself and its inclusion in core, but awaiting a working PBS. Perhaps that's not necessary, but it would feel a lot better to do things in that order, to me.

[docwilmot]

Still not serving correct paging, and I still don't know why 😢

[docwilmot]

The Project Browser requires jQuery UI MultiSelect Widget. To bring the PB to core, will we need to add this widget as a core library?

[quicksketch]

The Project Browser requires jQuery UI MultiSelect Widget. To bring the PB to core, will we need to add this widget as a core library?

Hm, could we avoid this dependency? Other options could be:

• Plain multi-select list.
• Plain checkboxes.
• Or checkboxes in a table with the core tableselect.js enhancement.

Clearly I need to try PB to see where this UI is being used.

I merged in the fixes for backdrop-ops/backdropcms.org#102 finally, so hopefully that will help paging.

[docwilmot]

Clearly I need to try PB to see where this UI is being used.

Clearly so do I, since I don't know. I should have checked first before posting.

[docwilmot]

The jQuery UI MultiSelect Widget was for project categories, which we dont have yet, until we start collecting more data from .info files. So removed. PR for PB against core coming up.

[klonos]

Couple of questions:

1. How does this currently look? Do we have a video demo or something available or are we blocked on implementing the server side?

[docwilmot]

@klonos https://github.com/backdrop-contrib/project_browser
Not much change in the merge to core, just moving the code to update module and adding tests. So install and test please, would be grateful for some full feedback.
The server has been fully functional (at last) for a few weeks since @quicksketch fixed query issue.

[klonos]

OK I just gave this a quick spin. I usually test every module that seems interesting/useful once it has an official release (I'm "watching" the entire contrib repo). I take the official releases (even alphas/betas) as the maintainer saying "OK, this is ready for initial testing". Guess that's why I've never gave it a go previously.

Anyways, I've spotted a few things and I'll file issues in the project issue queue. Main thing is that out of the box I got a "Showing 0 to 0 of 0." results 😞

[quicksketch]

Thanks for giving this a spin @klonos! I haven't had time to check it out, but I'm hoping to try soon!

[docwilmot]

Oops, sorry about that. I never updated the server to use Backdropcms.org. Pushed latest code; please re-install for changes. Thanks.