Fields always defined as string collection in app service

[jayfive]

Packages & versions
Umbraco 10.6.1
ExamineX.AzureSearch - 4.1.4
ExamineX.AzureSearch.Umbraco - 4.1.4

Hi,

On one of our Umbraco installations we've got an issue where we're setting field definitions to either be a long or full text type. This works fine locally but as soon as we deploy to an azure app service everything is indexed as a Collection(Edm.String).

Do you have any thoughts on what might be causing this or what I can do to debug the problem?

Thanks,

John

[Shazwazza]

Can you advise on how you are setting your field definitions?

Fields by default are Collection(Edm.String) unless explicitly specified.

[jayfive]

We've set these up in a way mentioned in your docs.

serviceCollection.Configure<AzureSearchIndexOptions>(global::Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName, options =>
            {
                options.FieldDefinitions.AddOrUpdate(new FieldDefinition("FieldName", AzureSearchFieldDefinitionTypes.FullText));
                ...
            });

It was very intermittent but would always work locally.

The bizarre thing is that it's now started to work correctly in azure. The only change we've made is the switch to the client's domain rather than using the *.azurewebsites.net one.

[Shazwazza]

Hi,

In what part of your code do you have that code above for configuring the fields? To me this issue sounds like some sort of race condition or that somehow that Configure block of code is not being called for some reason.

You could try to change that to PostConfigure instead of Configure.

Do you have any errors in your logs that might indicate a related issue?

You could also try configuring options like this so that you can get some log statements output:

services.AddSingleton<IConfigureOptions<AzureSearchIndexOptions>>(
    sp => new ConfigureNamedOptions<AzureSearchIndexOptions>("InternalIndex", options =>
    {
        var logger = sp.GetRequiredService<ILogger<Startup>>();
        logger.LogInformation("Configuring ExamineX options...");
        options.FieldDefinitions.AddOrUpdate(new FieldDefinition("FieldName", AzureSearchFieldDefinitionTypes.FullText));
    }));

[jayfive]

We have this code in the ConfigureServices method of Startup. Should it be moved to Configure(IApplicationBuilder app, IWebHostEnvironment env)?

There's nothing useful in the logs. I'll try sticking in some logging as you've suggested.

[Shazwazza]

Hi @jayfive, Did you try using the PostConfigure of configuring IOptions? You need to do this in ConfigureServices because you are configuring services in the DI container - namely the ExamineX IOptions.

[chrden]

@Shazwazza I am having a similar issue. My field definitions seem to be set intermittently, both locally and when hosted on IIS on an on-prem server. I can't confirm if this also happens on Azure App Services just yet.

I was previously using Configure and thought the issue had been resolved as my FieldDefinitions were correct in my Azure Search index for my local environment. However, when I deployed this to our test environment on our on-prem server, some of the FieldDefinitions were correct, others were not. I then deleted the local environment index, rebuilt and get the same issue on there again.

I have tried your suggestion above of using PostConfigure which seems to consistently resolve the issue locally, but not for our test environment.

I have also tried your logging code and can confirm that the FieldDefinitions are being picked up as the log entry appears but there are no logs suggesting any other issues.

My ConfigureServices in Startup.cs looks like this:

public void ConfigureServices(IServiceCollection services)
{
	var config = services
		.AddUmbraco(_env, _config)
		.AddBackOffice()
		.AddWebsite()
		.AddComposers()
		.AddCustomAppsettingSections()
		.AddCustomBackofficeSections()
		.AddCustomServices()
		.AddCustomContentFinders()
		.AddCustomConfigureOptions()
		.AddCustomComponents()
		.AddCustomNotificationHandlers();

    ...
}

The FieldDefinitions are in the AddCustomConfigureOptions extension method:

public static IUmbracoBuilder AddCustomConfigureOptions(this IUmbracoBuilder builder)
{
	builder.Services.PostConfigure<AzureSearchIndexOptions>(UmbracoIndexes.ExternalIndexName, 
		options =>
		{
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.HomeId, AzureSearchFieldDefinitionTypes.Integer));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Geolocation.Latitudes, AzureSearchFieldDefinitionTypes.Double));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Geolocation.Longitudes, AzureSearchFieldDefinitionTypes.Double));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Schema.PageTitle, AzureSearchFieldDefinitionTypes.Raw));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Sorting.CustomSortIndex, AzureSearchFieldDefinitionTypes.Integer));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Sorting.SortDateTime, AzureSearchFieldDefinitionTypes.DateTime));
			options.FieldDefinitions.AddOrUpdate(new FieldDefinition(ExamineConstants.Fields.Sorting.UpdateDateTime, AzureSearchFieldDefinitionTypes.DateTime));
		}
	);

	return builder;
}

This produces the following results when doing the following search in Azure Search in the Azure Portal:

{
  "search": "*",
  "select": "homeId,latitudes,longitudes,pageTitle,customSortIndex,sortDateTime,updateDateTime"
}

External index - Local

{
      "@search.score": 1,
      "sortDateTime": "2023-08-24T12:00:00Z",
      "latitudes": null,
      "longitudes": null,
      "pageTitle": "Your medical team ",
      "updateDateTime": "2023-08-24T12:00:00Z",
      "customSortIndex": 0,
      "homeId": 1057
}

External index - Test

{
      "@search.score": 1,
      "updateDateTime": [
        "24/08/2023 12:00:00 PM"
      ],
      "latitudes": null,
      "pageTitle": [
        "Your medical team "
      ],
      "customSortIndex": 0,
      "sortDateTime": [
        "24/08/2023 12:00:00 PM"
      ],
      "homeId": [
        "1057"
      ],
      "longitudes": null
}

The results are clearly different and it's as if the FieldDefinitions are not picked up when the same code runs on the Test environment.

I have tried deleting the index and rebuilding multiple times with the same results.

Can you suggest anything we could try?

Thanks

Thanks

[Shazwazza]

Hi @chrden are you also using any composers to modify your indexes? (i.e. lucene indexes?)

[chrden]

@Shazwazza We believe this may be related to race conditions. We will keep investigating and report back if necessary. Thanks 👍

[pariashiri]

Hello, we are experiencing a similar issue. We have a custom field which I defined like this:

public class ConfigureExamineIndexOptions : IConfigureNamedOptions<LuceneDirectoryIndexOptions>
    {
        public void Configure(string name, LuceneDirectoryIndexOptions options)
        {
            if (name != Constants.UmbracoIndexes.ExternalIndexName)
                return;

            options
                .FieldDefinitions
                .AddOrUpdate(new FieldDefinition("MyField", FieldDefinitionTypes.FullText));
        }
        public void Configure(LuceneDirectoryIndexOptions options)
            => Configure(string.Empty, options);
    }

I have deployed it to the server, in six websites, MyField index type is String but only one of them is Collection, and this makes my search broken on that site. Also locally, with the same database is String as well.

[Shazwazza]

Hi @pariashiri and others this discussion may be of help #87 (comment)

The main point is - if you are using Umbraco composers to customize LuceneDirectoryIndexOptions, then you must attribute your composer with [ComposeBefore(typeof(ExamineXComposer))]

The reason for that is because there is no guarantee the order in which Umbraco will find and load composers and if you are customizing LuceneDirectoryIndexOptions in a composer, then this customization must be done before the ExamineXComposer.

I will update the docs with regards to this too. Unfortunately there isn't a magic way for ExamineX composers to always run after your own. In future ExamineX versions, we will probably remove ExamineX from being composed by composers and move to extension methods that you need to opt-in for in your Startup.cs.

[pariashiri]

Thanks for the reply but, I don't use composer, I 'm adding ConfigureExamineIndexOptions to the startup class:
in this order:

 public void ConfigureServices(IServiceCollection services)
        {
            services.AddDataProtection();

            var umbracoBuilder = services.AddUmbraco(_env, _config)
                .AddBackOffice()
                .AddWebsite()
                .AddComposers()
                .AddSupplierNetwork()
                .AddEmbedProviders()
                .AddSlimsy()
                .AddDashboardCustomization();

            umbracoBuilder.ContentFinders().Append<SomeContentFinder>();

            // Load Balancing
            umbracoBuilder.SetServerRegistrar("....")["ServerRole"]);

            services.AddHttpContextAccessor();

            //ExamineX
            umbracoBuilder.Services.ConfigureOptions<ConfigureExamineIndexOptions>();
            services.AddSingleton<ExaminexExternalIndexFieldFilter>();
            services.AddSingleton<ExaminexInternalIndexFieldFilter>();
.
.
.

I am using Umbraco version 10.8.0-rc
Maybe I should try composing?

[pariashiri]

An update about the index problem,
I changed LuceneDirectoryIndexOptions to AzureSearchIndexOptions and it fixed the problem.

[Shazwazza]

@pariashiri if you are using extension methods to configure LuceneDirectoryIndexOptions and not a composer that is great! But the important part is to put that before .AddComposers(). This is because both Umbraco and ExamineX configure these via composers and your changes need to come first. ExamineX will map over LuceneDirectoryIndexOptions to AzureSearchIndexOptions in its composer, so your configuration of LuceneDirectoryIndexOptions needs to happen before ExamineX.

Alternatively, as you say, the safest bet is to use AzureSearchIndexOptions, but that just means you need to specify both LuceneDirectoryIndexOptions for working locally and AzureSearchIndexOptions for when working against azure search.

[pariashiri]

@Shazwazza Hi, after a while, my index was working great suddenly after an index rebuild, again some fields became collections, and my search is broken in live sites, last time I fixed it by changing LuceneDirectoryIndexOptions to AzureSearchIndexOptions but I have no clue how can I fix it now. we must fix this issue asap, do you have any suggestions?

[pariashiri]

I have changed my code to this but it is still same:

public static class ExamineXExtentions
    {
        public static IUmbracoBuilder AddCustomExamineIndexes(this IUmbracoBuilder builder)
        {
            builder.Services.PostConfigure<AzureSearchIndexOptions>(UmbracoIndexes.ExternalIndexName,
                options =>
                {
                    options.FieldDefinitions.AddOrUpdate(new FieldDefinition("IsCpd", FieldDefinitionTypes.FullText));
                    options.FieldDefinitions.AddOrUpdate(new FieldDefinition("SortDate", FieldDefinitionTypes.Long));
                    options.FieldDefinitions.AddOrUpdate(new FieldDefinition("WebsiteId", FieldDefinitionTypes.FullText));
                    options.FieldDefinitions.AddOrUpdate(new FieldDefinition("ArticleTypeName", FieldDefinitionTypes.FullText));
                }
            );
            return builder;
        }
    }

In start up

     var umbracoBuilder = services.AddUmbraco(_env, _config)
                .AddBackOffice()
                .AddWebsite()
                .AddCustomExamineIndexes()
                .AddComposers()
                ...

[Shazwazza]

@pariashiri As mentioned above, if you are using LuceneDirectoryIndexOptions then they should be configured before the .AddComposers() call. But you are using AzureSearchIndexOptions, you should put this after the .AddComposers() call. Also, you have not mentioned what ExamineX version you are using?

I've used your exact code snippets and I cannot replicate this. Restarting sites and rebuilding the indexes work 100% of the time with the correct field settings.

[Shazwazza]

That said, putting your code snippet above or after AddComposers won't matter since you are using PostConfigureOptions, I've tested and verified both ways with your code snippets.

[pariashiri]

@Shazwazza thanks for the response, my version is 4.1.8 and my Umbraco version is 10.8.2, the new code is working locally but not on live. My main point is, that before these changes it was working ok for almost 6 months, I hadn't changed anything related to the index but after an external index rebuild it started making collections. So what do you think you could advise, the problem is the live environment, any order works locally.

[pariashiri]

next attempt I am going to update to version 4.1.9 remove PostConfigure and move my extension before .AddComposer()
=> result:
all custom fields have become a collection!

[Shazwazza]

@pariashiri Definitely recommend using the latest version that you can to ensure that you have any bug fixes/enhancements made. That said, if you re-read above (sorry for any confusion), I would recommend that:

• You continue to use PostConfigure with AzureSearchIndexOptions (don't use LuceneDirectoryIndexOptions)
• You execute this code after AddComposer, you could put this anywhere really but I'd suggest putting it as the last part of the UmbracoBuilder chain

You should only use LuceneDirectoryIndexOptions if you need to run Examine locally with Lucene. If you do need to do this, then don't use a composer and put these before .AddComposer calls

[pariashiri]

@Shazwazza I don't have good news, still they are a collection and, as one of them is used for sorting, my sort is broken and I can't patch it, as collection can't be used for sorting. I post all my code:
startup:

    var umbracoBuilder = services.AddUmbraco(_env, _config)
        .AddBackOffice()
        .AddWebsite()   
        .AddComposers()                 
        .AddNotificationAsyncHandler<ContentPublishedNotification, xxx>()
        .AddNotificationAsyncHandler<ContentPublishedNotification, xxx>()
        .AddNotificationHandler<ContentPublishingNotification, xxx>()
        .AddNotificationHandler<ContentPublishingNotification, xxx>()
        .AddSupplierNetwork()
        .AddEmbedProviders()
        .AddSlimsy()
        .AddDashboardCustomization()
        .AddCustomExamineIndexes();

    umbracoBuilder.ContentFinders().Append<IssueContentFinder>();

    // Load Balancing
    umbracoBuilder.SetServerRegistrar(_config.GetSection("LoadBalancing")["ServerRole"]);

    services.AddHttpContextAccessor();

    services.AddSingleton<ExaminexExternalIndexFieldFilter>();
    services.AddSingleton<ExaminexInternalIndexFieldFilter>();

AddCustomExamineIndexes class:

  public static class ExamineXExtentions
  {
      public static IUmbracoBuilder AddCustomExamineIndexes(this IUmbracoBuilder builder)
      {
          builder.Services.PostConfigure<AzureSearchIndexOptions>(UmbracoIndexes.ExternalIndexName,
              options =>
              {
                  options.FieldDefinitions.AddOrUpdate(new FieldDefinition("IsCpd", FieldDefinitionTypes.FullText));
                  options.FieldDefinitions.AddOrUpdate(new FieldDefinition("SortDate", FieldDefinitionTypes.Long));
                  options.FieldDefinitions.AddOrUpdate(new FieldDefinition("WebsiteId", FieldDefinitionTypes.FullText));
                  options.FieldDefinitions.AddOrUpdate(new FieldDefinition("ArticleTypeName", FieldDefinitionTypes.FullText));
              }
          );

          return builder;
      }
  }

If you need to see ExaminexExternalIndexFieldFilter class let me know but they are filtering down what fields need to be indexed.

Umbraco version 10.8.2
ExamineX.AzureSearch.Umbraco Version: 4.1.9

If you run it will work locally but not in live env.

It's really frustrating, if there is a way to change an index after indexing or any fix I can do, please let me know.

[Shazwazza]

@pariashiri Unfortunately I cannot replicate. If it works locally but not in prod, I would urge you to try to determine what the difference is between environments and that the code you are using locally is in fact the code being used in prod. You will also need to ensure that you rebuild your existing indexes in prod when you publish the changes.

Also, there is some confusion over how you need to configure the fields that are sortable. These are the field types you can use: https://examinex.online/customization#default-field-types--analyzers

You are using FieldDefinitionTypes.FullText (or AzureSearchFieldDefinitionTypes.FullText)

The field will be indexed with the index’s default Analyzer without sortability. Generally this is fine for normal text searching.

... and will be created as a single string field and is not sortable.

If you want sorting, use FieldDefinitionTypes.FullTextSortable (or AzureSearchFieldDefinitionTypes.FullTextSortable)

Will be indexed with FullText but also enable sorting on this field for search results.

... and will be created as a single string field with the sortable property set to true.

You are creating the "SortDate" as FieldDefinitionTypes.Long - can I assume this field is being created correctly?