Nested structures for organising assets

[Sebobo]

It became apparent in many projects that the current tag and collection system doesn't cover all cases in which editors would like to organise their assets.
Other (DAM) systems also use folder like structures that can currently not be synced into Neos.

Often something like folders (similar to TYPO3) are requested.

The initial idea (see #3) was to implement nested tags. But when discussing the idea further more problems with this approach came up so the idea was dropped for now.
As tags can be assigned to any asset and more than one, thinking of them as folders doesn't really work well.

Collections were also meant for something else. As their can be assigned to sites and restricted via policies nesting them would also not solve our issues.

Therefore a new type Folder could be introduced that can be nested and could already be implemented in the Neos CR.
This way we could introduce them in parallel via the new Media UI without modifying the old UI or its API. At the same time this would be the first step to move everything into the CR asset related to have also translation and other capabilities in the future that are often requested.
Integrators could access this folder nodetree to already use it to access & reference assets.

A folder should have a label (translateable via the CR) and childnodes, those can be assets or more folders initially.
This can later be extended via NodeTypes.

[Sebobo]

@bwaidelich you said during our last call about this topic (last summer?) you wanted to write down an architecture idea. Maybe you can find some time in the next weeks to do this, would be awesome :)

[DrillSergeant]

@Sebobo Thanks for those ideas. In almost every Neos project our customers are missing a folder-structure to organize their assets. I also connected meanwhile different DAM-systems which had folder-structures. I am sure this feature would be loved by many customers.

[kdambekalns]

Working on an asset source integrating Canto right now and this topic came up as well.

Concepts – Neos vs Canto

• Tags are available in Canto, so could be mapped as-is from the asset source. They can not be nested in Canto, but are just plain-text, free-form things that can be input without any restrictions
• Canto has "Keywords" that are more like Neos tags, they need to be created explicitly by someone who has the needed permissions
• Canto has folders, those can be nested at will
• And Canto offers "custom fields" having a name and a set of values

The customer now would like to map those custom fields to asset collections and tags, mostly because that is what exists in Neos. And I guess we'll implement it like that somehow, and refactor later.

The future

My dream would be:

• map tags to tags
• have folders in Neos to map Canto folders from
• have some way to hook into some "faceted search" for the UI to use thos custom fields (on the Neos side those could be implemented as user-defined asset properties and "just be mapped from Canto")

Asset collections

And then someone needs to define what asset collections are to be used for "officially".

Their behaviour (in relation to tags) has been explained in https://discuss.neos.io/t/media-module-concept-of-tags-and-collections/3158 and that still is true. And I'd say they can be used to "group by purpose", i.e. group all employee photos, all screenshots, …

At least as long as folders do not exist. Then the collections could serve a different purpose, but what should that be, exactly?

[tdausner]

Why don't utilize collections having a nested structure?
Example collections

site1/
site1/usergroup1
site1/usergroup1/purpose1
site1/usergroup1/purpose2
site1/usergroup1/purpose2/subset1
site1/usergroup2
site1/usergroup2/purpose1
site1/usergroup2/purpose2
site2/
...

The separation character could be defined in a yaml file.

This nested collection structure could be displayed in a "folder tree"

site1
  usergroup1
    purpose1
    purpose2
      subset1
  usergroup2
    purpose1
    purpose2
site2
...

Collection titles are defined as varchar(255) giving the opportunity for deep nesting.

To achieve simple site dependent restricted access to the asset collections the privilegeTargets matchers are extended (a job I've already done but still under test):

privilegeTargets:

  'Neos\Media\Security\Authorization\Privilege\ReadAssetCollectionPrivilege':
    'First.Site:AssetCollection':
      label: 'First site asset collection access'
      matcher: 'titleStartsWith("site1/")'
    'Second.Site:AssetCollection':
      label: 'Second site asset collection access'
      matcher: 'titleStartsWith("site2/")'

  'Neos\Media\Security\Authorization\Privilege\ReadAssetPrivilege':
    'First.Site:AssetRead':
      label: 'First site asset access'
      matcher: 'isInCollectionStartsWith("site1/")'
    'Second.Site:AssetRead':
      label: 'Second site asset access'
      matcher: 'isInCollectionStartsWith("site2/")'

with the corresponding roles:

roles:
  'First.Site:FirstAssets':
    label: 'First site asset access'
    description: 'Access to First-Assets'
    parentRoles: ['Neos.Neos:Editor']
    privileges:
      -
        privilegeTarget: 'First.Site:AssetCollection'
        permission: GRANT
      -
        privilegeTarget: 'First.Site:AssetRead'
        permission: GRANT

  'Second.Site:SecondAssets':
    label: 'Second site asset access'
    description: 'Access to Second-Assets'
    parentRoles: ['Neos.Neos:Editor']
    privileges:
      -
        privilegeTarget: 'Second.Site:AssetCollection'
        permission: GRANT
      -
        privilegeTarget: 'Second.Site:AssetRead'
        permission: GRANT

(don't forget to give administrator & asset manager the same privileges)

IMHO it's always best to reuse proven code as the NEOS privilegeTarget system is. NEOS wins a "folder structure" by a backend visualization task (in gridLeft and gridRight). And the new mechanism of tagging still coincides with such a nested collection nested structure

[Sebobo]

@tdausner I would rather not force a new feature onto a confusing existing feature. It would also mean, that we also have to convert the collections names back and forth and find a way to handle "missing" collections in the path. It then depends so much on how the integrator configures everything.

I currently see the most benefits in having everything in the CR at some point. For which we also have privileges. Then we can also have unified permission checks for the Content module and the media ui.

[lorenzulrich]

After giving it some thought, I would be in favor of having a new concept called Folders. As stated in this post, asset collections have a lot of implications with other functionality such as tags, but also regarding permissions. E.g. if two asset collections are assigned to an asset, the editor needs to have permissions to both asset collections etc..

Regarding multi-site instances, I would not couple the concept too tightly to sites but leave it up to the integrator to set the necessary privileges independent of the site. You could have a folder with assets that should be available to all sites, a folder with site-specific assets etc..

[tdausner]

@Sebobo CR... so you think about a "folder" node tree /assets similar to /sites? Then some code from neos-ui/pageTree could be applied.

[tdausner]

@lorenzulrich "a new concept called Folders" -> something actually not present in NEOS or utilize an existing concept?

[lorenzulrich]

@tdausner Yes, I'm talking about a new concept. Basically what also @kdambekalns is suggesting in his post.

[Sebobo]

@tdausner yes, we want to have all asset related data in the CR at some point in a separate tree independent of the sites.
That would allow custom asset nodetypes, folders, tags, using references etc.
There is already https://github.com/neos/metadata-contentrepositoryadapter which I would either build upon or try to make it compatible.

[tdausner]

@Sebobo had a look at https://github.com/neos/metadata-contentrepositoryadapter. As the CR mapping is for assets it makes not too much sense to have dimensions along with. Unlike folders to hold assets in new-media-ui having natural language folder names.

@lorenzulrich here is a concept for a CR based folder system

Concept for a CR based folder system

Content Repository

Persons familiar with the Neos/Flow Content Repository may skip this section.

The Neo/Flow Content Repository (CR) is an abstraction layer to whatsoever is implemented as permanent storage
(usually a database system). This brief introduction does not claim completeness. It covers the aspects
required for the understanding of the CR based folder system. For a detailed documentation see [1].

The basic unit in the CR is a Node. Each Node is identified by

• path example: /var/www/html [2] - you know this from directory trees, or
• identifier example: c255046d-4215-456f-b2f9-017db059cf55 [3]

Any Node in the CR can hold properties. A property has a key and a value.

But that's not all. The CR can hold the same Node in different variations for example to keep different
properties for different languages and/or countries. Within the CR the different variations are called dimensions.

For one Node path the CR can hold variations (and hence different property values or properties) for any
combination of dimensions. Standard NEOS dimensions are language and country.

As of this a Node is unique by path or identifier AND dimensions.

Dimensions are defined in file Settings.yaml in Neos.ContentRepository.contentDimensions section (see [1] and
site package Neos.Demo). In the settings file there are defined constraints as not all combinations
of the defined dimensions (language and country) do make sense. You can define any dimension in
your project's Configuration/Settings.yaml file.

Neos Folders

For certain purposes there is a demand to store information permanently in a tree structure. This is achieved by
mapping Neos Folders to the CR's Node path tree.

Base path

To determine between different folder trees the first level of path is utilized as the basePath.
NEOS' sites are stored in the CR having a basePath named /sites.

Segments

On explode of a path by the slash / character the resulting array keeps the segment names:

$segmentNames = explode('/', '/var/www/html');
\Neos\Flow\var_dump($segmentNames);

Output:

Flow Variable Dump
array(4)
 integer 0 => string "" (0)
 integer 1 => string "var" (3)
 integer 2 => string "www" (3)
 integer 3 => string "html" (4)

Variants

Neos Folders also do utilize the dimensions feature of the CR. On creation of a variant all folder path names
are kept.

Title

At CR level a Node name is equal to a segment name. The Node with path /sites has the node name sites.
This is not suitable for folders, as the folder name shall be displayed to the backend user and the folder name must
not strictly follow the segment name. That's why a Neos Folder has a default property title to keep the display
name of a Neos Folder.

Use Case

You'll start with folders for dimension laguage=en_US. A simple folder tree contains

 /fruit
  ∟ /apples
  ∟ /pears

Next a new dimension language=de is added. Resulting an identical folder tree having dimension language=de
exists. For user handling the folder names shall be renamed. CR Node names must contain letters a-z, numbers 0-9
and the minus character [^2]. Umlauts and upper case characters cannot be mapped to Node names.

 language=en_US            language=de
 
 /fruit           <->      /Früchte
  ∟ /apples       <->       ∟ /Äpfel
  ∟ /pears        <->       ∟ /Birnen

Associations

A Neos Folder is intended to keep information. Primarily information of objects stored in a Neos Folder. To fulfill
this intention a Neos Folder has a default property associations to keep unique identifiers (an array of).
This can be any valid identifier [3].

API

Operations

All operations shall be dimension dependent.

Neos Folder operations shall be

• add a folder
• adopt a folder (new dimensions)
• remove a folder
• set folder title
• move a folder
• link (associate) a folder to some object(s)
• unlink (remove association) at a folder
• get folder tree

Command Line Interface

All operations shall be made available as flow commands

Services

All services shall be made available as Neos routes

Acknowledgements

Thanks to all the people inventing and programming the Flow framework giving people like me the
base for this concept and the subsequent implementation.

---

[1]: NEOS Content Dimensions documentation

[2] path regular expression ^(\/([a-z0-9](-[a-z0-9])*)*)+$

[3] identifier regular expression ^([a-f0-9]){8}-([a-f0-9]){4}-([a-f0-9]){4}-([a-f0-9]){4}-([a-f0-9]){12}$

[Sebobo]

@tdausner I would do everything the same way Neos does with pages. Folders have a title and the nodename in the CR is completely hidden and only a technical detail that the CR manages on its own. Therefore the title of the folder can also contain any UTF8 character like it works with Neos Documents.

The entry node path could be /media/folders. We probably need some listeners to make sure that the folder tree doesn't not change between dimensions and only their titles.

Both assets and folders get advantages from the CR and its dimensions as both contain titles and other metadata that can and should be localisable.

[tdausner]

@Sebobo changed folderName to title. Added "On creation of a variant all folder path names are kept." to Variants section.
"some listeners": the API is intended to keep the folder path structure. Rename affects title only. What's your intention to establish "some listeners"?

[bwaidelich]

my 2ct re CR based assets: IMO we should base that on the new (event sourced) content repository because it offers a couple of features that will make it really hard otherwise.
For example will it be possible to have a dedicated CR for the assets that doesn't mix in node types and dimensions from the rest of the website.
Also, if we use the node hierarchy as folders, we need a second dimension, e.g. to find assets by referenced collections or tags. With the new CR references are always bidrectional so that this should be possible without having to introduce yet another index that can get out of sync

[lorenzulrich]

And my two cents to add to it: While I agree it would be nice to build it on top of the new CR, this would exclude it from working with Neos 8. I don't think it would be wise to wait until Neos 9 with this feature because for big sites, not having nested folders is kind of a showstopper.

[tdausner]

@bwaidelich @lorenzulrich thanks for your 4ct ;-)

@bwaidelich For this special purpose (folders in the new-media-ui) it's a demand to have dimensions. I have started an implementation based on Neos\ContentRepository\Domain\Model, Neos\ContentRepository\Domain\Repository and Neos\ContentRepository\Domain\Service (the latter for context and dimensions). I didn't follow the ES-CR details so I wonder if the mentioned resources shall be obsolete with Neos 9 and ES_CR?

@ALL I agree with @lorenzulrich to develop on the new-media-ui base Neos 8.

[bwaidelich]

For this special purpose (folders in the new-media-ui) it's a demand to have dimensions

Right, but not (necessarily) the same ones as all the other sites in the installation.

I don't think it would be wise to wait until Neos 9

Neos 9 is scheduled for August this year. And while it's quite possible that it won't be finished on time, I'm afraid that adding more features on an by then outdated API will slow down the process even more.

this would exclude it from working with Neos 8

I don't think so. If it is properly encapsulated and expose a dedicated PHP API, it could be installed alongside the current CR I assume – probably not out of the box, but we should be able to get there anyways in order to increase traction

[Sebobo]

My plan is to implement neos/metadata support and the first folder implementation in March if the sponsoring works out.

It might be behind a feature flag like some other features. I'm not sure yet whether I want to have yet another separate package as it puts quite a strain on maintenance. But with migrations etc. I don't see that as a big problem right now. The first goal is to improve usability in the media management and not providing an amazing folder system for everything Neos. And I wouldn't connect tags and collections yet.

[mficzel]

I like the idea but would suggest to use the base path /media instead of /media/folders as this will allow to put the metadata as nodes into the folders at a later time without need to move folder-nodes around.

This also imho goes well with concepts of the new cr as it would be a root node „media“ that can be put into any cr as required. It might even make sense at a later time to create other media libraries with other root nodes and different access rights/dimensions etc. In that case „media“ would be the default name of the primary media library. Obviously this is far away from now.

However I would also really like to see this on Neos 8 and start using it :-)

[tdausner]

Based on the concept I've implemented a Neos.Folder package. Please test from https://github.com/tdausner/Neos.Folder.

@Sebobo IMHO you could start with this package. When you intergrate Neos.Folders into new-media-ui please consider a folder level for the site name: /media/folder/siteName or /media/siteName/folder. To simplify media collection & tag storage, I'd propose a solution utilizing the Neos.Folder system:

• /media/siteName/collections/<collection name>
• /media/siteName/tags/<tag name>

The folders <collection name> and <tag name> can hold the associations between an asset and a collection/tag

Looking forward to your comments and improvement proposals ;-)

[Sebobo]

Hi @tdausner oh wow. Will have to look into it.

Currently I would define the starting point for the folder nodes via a setting to allow for later changes. Then it's easy to fix the path with a migration if necessary. I like @mficzel suggestions with /media/folders

I wouldn't include anything about sites into the structure at this point as the media system doesn't know anything about them and currently I don't see why it should. Editors can solve that themselves with folders or integrators can use privileges in the future.

As I already said, tags and collections are a future step which I will not include for now. They are also not connected to sites currently. Probably they will go into /media/tags and /media/collections respectively.

So my first step is to build the UI part with the integrated dev-server, then the GraphQL part with a dummy folder interface and implementation. The last step would be the actual folder integration into the CR.
It might already make sense to create some flexibility here for custom implementations via Objects.yaml override and connected to the Asset Sources. There are asset sources which would be able to supply the UI also with (readonly) folders.

[tdausner]

@Sebobo:

I wouldn't include anything about sites into the structure at this point as the media system doesn't know anything about them and currently I don't see why it should. Editors can solve that themselves with folders or integrators can use privileges in the future.

My last job at CGM in Koblenz was the CMS switch to NEOS together with punkt.de (@daniellienert). The installation is multi site multi dimensions. We had the problem with licensed assets, as CGM has many independent subsidiaries, and licensed assets are licensed for one subsidiary (mostly of peculiar reasons).

Another point of view is a scenario, where a service provider wants to run a single NEOS installation with multi sites for different customers. The asset manager of one site must not have access to assets of other sites. Not only to licensed assets (images) but also to (PDF) corporate information.

So please consider a "license" or "corporate" folder level configurable by a NEOS administrator (or in some yaml configuration).

So my first step is to build the UI part with the integrated dev-server, then the GraphQL part with a dummy folder interface and implementation. The last step would be the actual folder integration into the CR.

If you plan to connect the media-ui to Neos.Folder by web service this can be easily performed. All the functionality you've listed in #179 can be solved by the Neos.Folder web services.

• Tree view - Get folder tree
• Add folder - Add a folder to parent folder identified by token*
• Edit folder - Set title at folder identified by token*
• Remove folder - Remove folder identified by token*
• Assign Asset to folder - Set or clear association
• Filter assets by selected folder (granted by associations)

* IMHO the term token to identify a folder fits best, as a folder token can either be a node path or an identifier

[tdausner]

To give it a try:

./flow folder:add '/fruit/apple/European Elstar' '' language=en_US --recursive
./flow folder:add '/fruit/apple/Prince Albrecht of Prussia' '' language=en_US
./flow folder:add "/fruit/pear/Willam's christ" '' language=en_US --recursive
./flow folder:add "/fruit/pear/Bosc pear" '' language=en_US

./flow folder:adopt /fruit '' language=de --recursive

./flow folder:title /fruit Früchte language=de
./flow folder:title /fruit/apple Äpfel  language=de
./flow folder:title /fruit/pear Birnen language=de
./flow folder:title /fruit/apple/european-elstar Elstar language=de
./flow folder:title /fruit/apple/prince-albrecht-of-prussia 'Prinz Albrecht von Preußen' language=de
./flow folder:title /fruit/pear/bosc-pear 'Boscs Flaschenbirne' language=de
./flow folder:title /fruit/pear/willam-s-christ "Williams Christ" language=de

./flow folder:list /fruit language=en_US\&language=de

[tdausner]

Now registered in packagist.org:

composer require tdausner/neos.folder dev-origin

[Sebobo]

@tdausner

My last job at CGM in Koblenz was the CMS switch to NEOS together with punkt.de (@daniellienert). The installation is multi site multi dimensions. We had the problem with licensed assets, as CGM has many independent subsidiaries, and licensed assets are licensed for one subsidiary (mostly of peculiar reasons).

Another point of view is a scenario, where a service provider wants to run a single NEOS installation with multi sites for different customers. The asset manager of one site must not have access to assets of other sites. Not only to licensed assets (images) but also to (PDF) corporate information.

So please consider a "license" or "corporate" folder level configurable by a NEOS administrator (or in some yaml configuration).

I understand the request, but this is a bit much for me right now to consider, as this is more a privilege problem than a node organisation problem IMO.
Whether an asset is available in a site is not defined by the folders, so your request is more like having a separate folder structure for each site. But as I planned to make the entry point configurable this could already be solved as one could go one level deeper in the folder structure in the Settings for a specific site. For this to work it has to have a separate Settings file of course.
And with the privileges you can prevent editing other trees than the current one.
Another way would be to use the FolderSource extensibility to provide your own source and there you can just return the folder subtree matching the users permissions.
Again whether an asset then shows up in the media module is defined by privileges and not the folders.

So is there really more needed for the first version?

[tdausner]

But as I planned to make the entry point configurable this could already be solved as one could go one level deeper in the folder structure in the Settings for a specific site.

This and folder privileges should meet all needs (hopefully) . :-)

[Sebobo]

So after a deeper dive into the existing Neos.Media code, thinking about the UX and a longer discussion with @bwaidelich we decided to first allow collections to have a parent therefore making them hierarchical.

This brings us already most of the internal functionality and existing projects can easily be adapted.

I was already able to make the basic works as it's visible in the screenshot.

We now think that it would be easier to think about a CR integration with the new CR and with an API rewrite currently targeted for Neos 10.

After I had added a new folder tree I wasn't happy with the combination of collections, tags and folders. So having just the collection tree and changing the meaning of tags for collections to actually tagging collections instead of defining what tags you are allowed to add to assets in a collection makes much more sense IMO. Therefore I would also remove tags from the tree and show them in a different way (list?maybe I will also hide the technical collection label and call everything folders in the UI.

Currently I feel quite good with the solution even though we thought it wouldn't be so good.

I also plan to support nested collections if provided by asset sources.

So we can also one day have a Neos CR asset source with all the bells and whistles.

@tdausner sorry it took me a day of implementing and deleting everything again to understand the road blocks we currently have. Even though I absolutely don't rule out the CR way for the not super far future.

Wdyt?

[lorenzulrich]

@Sebobo Thanks for your update. In general, I don't think having nested collections is a bad idea.

However, in practise I see a problem with multi-site instances: As recommended, we are normally using one collection per site (which now could have child collections - which is great) and then using AssetCollection privileges so they editor only sees the allowed collections.

But the UI allows adding an asset to more than one collection - in contrast to folders, where a file always resides in one folder (at least that's the way it is usually integrated). So an editor who has access to site A and B might add an asset to two collections of two different sites. But an editor only having access to site A cannot edit/delete/replace this asset because of insufficient permissions. With folders, this would not happen.

Maybe the UI could be made configurable so a file can be added to one collection only (although the API allows for more than one). Then I think (depending on the "edit asset UI" which is really messy with a lot of collections in the old Media module) our immediate needs would be covered.

[Sebobo]

Hi @lorenzulrich thx for the feedback. Exactly that was the idea we discussed. Having a feature toggle to only allow one collection per asset.

[bwaidelich]

I'm really happy that we don't introduce yet another concept with no clear distinction.
The concern of having assets in multiple collections is real though, but I think a global feature toggle is a good compromise.
I'd say with nested collections with a clear 1:n relation to assets and tags with the n:n relation most usecases should be covered – so I would even suggest do encourage the usage of this flag by default

[tdausner]

I'm following the discussion from hospital (total knee arthroplasty left). To get to a final demand coverage analysis: has anybody put down a collection of use cases?

[Sebobo]

Hi, I mostly posted screen and updates of this on Mastodon, but I forgot to update here too:

Here is the normal tree view with my latest changes:

The little yellow stars are the new favourites feature.
The grey collections are the ones without any assets, but I think about not using color but instead maybe a stacked icon or something. Showing (0) after all their labels looked bad.

The favourites can also be filtered with the header that is now a dropdown:

This view does not show nested collections as this would create some problems when nested collections are favourites too. Also it should be some kind of focus mode.

The collection inspector now shows a breadcrumb and a parent collection selector, both are work in progress:

Besides that I'm trying to add some new asset Eel helpers to 8.3 in neos/neos-development-collection#4155 to make it easier to render the assets in a collection or tag.
I plan to add datasources in this package to allow integrators to combine the selection of collections and tags with the display of their contained assets.

And here is the first draft of adding the parent/child relation to 7.3 to make the hierarchical collections possible without AOPing it into the domain models neos/neos-development-collection#4160

@tdausner I hope you are better and at home again!

Regarding the main use cases I focus on are:

1. give editors a way to organise their assets and make them easier to find again.
2. make it easier to implement galleries and other elements that show assets by collection and tag (see my Eel helper PR and notes)
3. make it easier to implement policies on multiple collections (which I haven't implemented yet)
4. help projects with a large amount of collections which might not be nestable (see favourites)
5. support folders from asset sources (not started anything related to that yet)
6. make usage of collections and tags less confusing than its now in the media browser

I'm sure there are more complex demands from DAM systems, but there are a wide range of ways to get those implemented too and I think we have to talk about extensibility instead of trying to push everything into this package.

[tdausner]

The grey collections are the ones without any assets, but I think about not using color but instead maybe a stacked icon or something. Showing (0) after all their labels looked bad.

Showing (0) or something similar is essential - keep visual disabled people in mind. I like the greyed out empty collections.

I hope you are better and at home again!

yepp!

Regarding the main use cases I focus on are...

To get a "todo/done" list the use cases should be more precise

• as an editor I shall organize my assets in a hierarchical structure
• as an editor all assets of a gallery shall be filtered by a collection and/or a tag
• as an editor I shall create tags and/or collections
• as an asset manager I shall have the tool for setting policies on collections
• and so on

IMHO the media-ui should be kept as simple and small as possible. Possibly it's not always the best to create a jack of all trades (eierlegende Wollmilchsau). Leave DAM functionality at DAM.