Add ability for the main Umbraco search to include media file names #6579

Jeavon · 2019-10-04T09:47:35Z

The PR adds a new field to the MediaValueSet called "__umbracoFile", "__" prefixed as this is to be used internally only.
The new field is populated either by extracting the Src property from the Image Cropper Json or directly from the property in the case of files such as PDFs.
We have also updated the UmbracoTreeSearcher to include the new field and modified the mapper to include the filename in the search results in braces.

…l as node name.

bjarnef · 2019-10-04T09:58:36Z

@Jeavon should we add this as a constant here?
https://github.com/umbraco/Umbraco-CMS/blob/v8/dev/src/Umbraco.Examine/UmbracoExamineIndex.cs

and then use it like this similar to UmbracoExamineIndex.NodeKeyFieldName:

{UmbracoExamineIndex.UmbracoFileFieldName, new object[] {umbracoFile}}

if (source.Values.ContainsKey(UmbracoExamineIndex.UmbracoFileFieldName))
{
    var umbracoFile = source.Values[UmbracoExamineIndex.UmbracoFileFieldName];
    if (umbracoFile != null)
    {
        target.Name = $"{target.Name} ({umbracoFile})";
    }
}

Jeavon · 2019-10-04T10:08:09Z

@bjarnef yes we should, will add it

Shazwazza

I don't think the field needs to be prefixed with __ generally that indicates it's a system field that isn't analyzed and can only be exact matched. Have left a couple other comments too.
Cheers!

src/Umbraco.Examine/MediaValueSetBuilder.cs

src/Umbraco.Web/Search/UmbracoTreeSearcher.cs

Jeavon · 2019-10-04T17:48:19Z

@Shazwazza trying to come up with a Lucene query that will allow for trailing wildcard but where symbols (hyphens, fullstops, undersores) are allowed, any ideas.....
For example, a file called "jancaps-copy.jpg" Then this query doesn't work +(umbracoFileSrc:jancaps\-copy* )

Shazwazza · 2019-10-08T03:32:51Z

@Jeavon don't worry about searching on Key, i've added that in a different PR, see #6530

As for your question regarding escaping chars and allowing wildcards, can you make anything work in Luke? Does the wildcard query work when you aren't escaping chars? Is it because the hyphens gets stripped out by the standard analyzer and for this field it should use whitespace analyzer?

Jeavon · 2019-10-09T11:47:18Z

@Shazwazza I spent quite a while in Luke, the only thing I could make work was to replace the hyphens with spaces +(umbracoFileSrc:jancaps copy.jpg* ) this then works perfectly.
I believe the internal index is using the CultureInvariantWhitespaceAnalyzer so I don't think that's an issue...

Jeavon · 2019-10-09T15:35:51Z

I've added a workaround to replace the hyphens to spaces when searching this field. It now works as expected and shown in this gif however maybe there is a better way that I have not found...

Shazwazza

Hi @Jeavon, found a few other potential changes/questions/comments. Regarding searching with hyphens and wildcards, i found this SO thread: https://stackoverflow.com/questions/16858880/java-lucene-search-query-hyphens-with-wildcards

Could be another more effective work around: https://stackoverflow.com/a/36320124/694494)

Else the correct way to do it would be to use a custom analyzer for this field that strips out hyphens

Shazwazza · 2019-10-17T03:57:34Z

src/Umbraco.Examine/MediaValueSetBuilder.cs

+
+                if (!string.IsNullOrEmpty(umbracoFilePath))
+                {
+                var uri = new Uri(_runtimeState.ApplicationUrl.GetLeftPart(UriPartial.Authority) + umbracoFilePath);


Do we need to use the application URL here? i mean, all this is really doing is constructing a Uri just so that it parses, you could just as easily have a static URI with a dummy hostname, etc... so you aren't allocating a new object for this each time.

I could have a dummy URI but that felt somehow wrong (I could also split the string), but can do that, what would you suggest for a dummy URI (is there a precedent)?

I think a dummy URL would be better and then we don't need to inject the IRuntimeState, at the end of the day it won't make any difference and we don't need to change as much

@Jeavon this will be the last thing we need to cleanup i think and then we can merge

@Shazwazza updated, thanks!

src/Umbraco.Tests/UmbracoExamine/IndexInitializer.cs

src/Umbraco.Web/Search/UmbracoTreeSearcher.cs

Jeavon · 2019-10-17T16:56:37Z

I like the ? idea, I'll try that also

Jeavon · 2019-10-29T15:35:46Z

@Shazwazza
I switched to using a List as suggested.
I have left you a comment about the Url here https://github.com/umbraco/Umbraco-CMS/pull/6579/files#r336117246
I tried the ? suggestion for the wildcard search but it didn't work (in Luke also), space seems to work perfectly so leaving as that....?

Shazwazza

Couple questions left inline

Shazwazza · 2019-11-05T00:36:11Z

src/Umbraco.Examine/MediaValueSetBuilder.cs

+
+                if (!string.IsNullOrEmpty(umbracoFilePath))
+                {
+                var uri = new Uri(_runtimeState.ApplicationUrl.GetLeftPart(UriPartial.Authority) + umbracoFilePath);


I think a dummy URL would be better and then we don't need to inject the IRuntimeState, at the end of the day it won't make any difference and we don't need to change as much

src/Umbraco.Web/Search/UmbracoTreeSearcher.cs

…r review

Shazwazza

Sorry @Jeavon just a couple small tweaks, sorry for being such a pain ;)

src/Umbraco.Examine/MediaValueSetBuilder.cs

src/Umbraco.Web/Models/Mapping/EntityMapDefinition.cs

…zation issues and therefore needing to mock the logger for the tests

Shazwazza · 2019-11-14T01:44:17Z

Awesome stuff @Jeavon all merged in 🎉

Jeavon added 2 commits October 3, 2019 17:46

Add ability for editors to search for media items by file name as wel…

e4df2f4

…l as node name.

Fixing Examine unit tests

72147ab

Adding a constant for the new field name

68866e9

Shazwazza suggested changes Oct 4, 2019

View reviewed changes

src/Umbraco.Examine/MediaValueSetBuilder.cs Outdated Show resolved Hide resolved

src/Umbraco.Web/Search/UmbracoTreeSearcher.cs Outdated Show resolved Hide resolved

Jeavon added 2 commits October 4, 2019 17:44

Rename new field and ensure Id is also searchable

7130c99

Add null check to umbracoFilePath and used Yield extension method

61465ed

Added a fix to handle hyphens in file names

82af5f2

Also replace underscores

0c957c2

Jeavon requested a review from Shazwazza October 14, 2019 09:27

Shazwazza suggested changes Oct 17, 2019

View reviewed changes

Jeavon added 2 commits October 29, 2019 12:13

Merge branch 'v8/dev' into v8/feature/search-media-files

0400487

Changed field to a List as suggested in review

76bbf01

nul800sebastiaan mentioned this pull request Oct 29, 2019

V8 : Need ability to search images and files by their real names #6937

Closed

nul800sebastiaan added the state/sprint-backlog label Oct 29, 2019

Shazwazza removed the state/sprint-backlog label Nov 5, 2019

Shazwazza suggested changes Nov 5, 2019

View reviewed changes

Jeavon added 2 commits November 12, 2019 10:12

Merge branch 'v8/dev' into v8/feature/search-media-files

b30f7b6

Switched to using a dummy Uri so that runtimeState isn't needed as pe…

10622cc

…r review

Jeavon requested a review from Shazwazza November 12, 2019 10:59

Shazwazza suggested changes Nov 12, 2019

View reviewed changes

src/Umbraco.Examine/MediaValueSetBuilder.cs Outdated Show resolved Hide resolved

src/Umbraco.Web/Models/Mapping/EntityMapDefinition.cs Outdated Show resolved Hide resolved

Code tweaks from PR review - also led to adding logging for Deseriali…

88f6dde

…zation issues and therefore needing to mock the logger for the tests

Jeavon requested a review from Shazwazza November 13, 2019 12:32

Shazwazza merged commit 88f6dde into umbraco:v8/dev Nov 14, 2019

Shazwazza added release/8.5.0 type/feature labels Nov 14, 2019

kjac mentioned this pull request Dec 13, 2019

V8: Don't append empty file names to search results #7331

Merged

1 task

clausjensen added release/8.4.0 and removed release/8.5.0 labels Jan 7, 2020

umbrabot added the type/feature label Oct 22, 2020

ronaldbarendse mentioned this pull request Oct 29, 2020

Umbraco Examine indexes contain duplicate fields and wrong field definitions #8964

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability for the main Umbraco search to include media file names #6579

Add ability for the main Umbraco search to include media file names #6579

Jeavon commented Oct 4, 2019 •

edited by Shazwazza

bjarnef commented Oct 4, 2019 •

edited

Jeavon commented Oct 4, 2019

Shazwazza left a comment

Jeavon commented Oct 4, 2019

Shazwazza commented Oct 8, 2019

Jeavon commented Oct 9, 2019

Jeavon commented Oct 9, 2019

Shazwazza left a comment

Shazwazza Oct 17, 2019

Jeavon Oct 17, 2019

Shazwazza Nov 5, 2019

Shazwazza Nov 11, 2019

Jeavon Nov 12, 2019

Jeavon commented Oct 17, 2019

Jeavon commented Oct 29, 2019

Shazwazza left a comment

Shazwazza Nov 5, 2019

Shazwazza left a comment

Shazwazza commented Nov 14, 2019

Add ability for the main Umbraco search to include media file names #6579

Add ability for the main Umbraco search to include media file names #6579

Conversation

Jeavon commented Oct 4, 2019 • edited by Shazwazza

bjarnef commented Oct 4, 2019 • edited

Jeavon commented Oct 4, 2019

Shazwazza left a comment

Choose a reason for hiding this comment

Jeavon commented Oct 4, 2019

Shazwazza commented Oct 8, 2019

Jeavon commented Oct 9, 2019

Jeavon commented Oct 9, 2019

Shazwazza left a comment

Choose a reason for hiding this comment

Shazwazza Oct 17, 2019

Choose a reason for hiding this comment

Jeavon Oct 17, 2019

Choose a reason for hiding this comment

Shazwazza Nov 5, 2019

Choose a reason for hiding this comment

Shazwazza Nov 11, 2019

Choose a reason for hiding this comment

Jeavon Nov 12, 2019

Choose a reason for hiding this comment

Jeavon commented Oct 17, 2019

Jeavon commented Oct 29, 2019

Shazwazza left a comment

Choose a reason for hiding this comment

Shazwazza Nov 5, 2019

Choose a reason for hiding this comment

Shazwazza left a comment

Choose a reason for hiding this comment

Shazwazza commented Nov 14, 2019

Jeavon commented Oct 4, 2019 •

edited by Shazwazza

bjarnef commented Oct 4, 2019 •

edited