Extension for Gridify to be able to use it with Elasticsearch

[ne4ta]

Description

Added an extension for original Gridify to work with Elasticsearch. The extension supports filter and sorting conversion to Elasticsearch DSL language queries.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

• I have performed a self-review of my code
• I have added tests that prove my fix is effective or that my feature works
• I have made corresponding changes to the documentation
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

[what-the-diff]

PR Summary

• New .gitignore file added
  A new .gitignore file has been created that ensures files of a certain format (/.*user) are not tracked by Git, preventing them from being uploaded to the repository and cluttering it with unnecessary files.

• Updated README.md
  The primary documentation file README.md has been improved by removing unnecessary line breaks and adding important information about compatibility with the Elasticsearch technology.

• New projects added to gridify.sln
  Two new components/projects, Gridify.Elasticsearch and Gridify.Elasticsearch.Tests, have been incorporated into the overall solution. This expands the solution's capability, especially in relation to Elasticsearch.

• Added necessary files for new project Gridify.Elasticsearch
  The project required ExpressionExtensions.cs, Gridify.Elasticsearch.csproj, GridifyExtensions.cs, and ToElasticsearchConverter.cs files to be introduced. These files handle distinct functionalities like working with expressions using extension methods, storing project configuration, handling Gridify in relation with Elasticsearch, and converting Gridify expressions into Elasticsearch queries, which boosts the adaptability and usability of the project.

• Enhancements to GridifyExtensions.cs & GridifyMapperConfiguration.cs files
  The GridifyExtensions.cs file now contains added internal static extension methods for further refining the handling of Gridify. Meanwhile, the GridifyMapperConfiguration.cs file gets a new public property, enabling additional customization options related to Elasticsearch naming conventions.

• New testing files for project Gridify.Elasticsearch.Tests
  For the new Gridify.Elasticsearch.Tests project, Gridify.Elasticsearch.Tests.csproj and TestClass.cs files have been introduced, empowering more detailed and efficient testing processes for the new Elasticsearch functionality.

[ne4ta]

I will update the documentation in the docs folder if you will be ready to extend your lib with my extension.

[alirezanet]

Hi @ne4ta,
Awesome job 🚀👏!
This looks really cool and useful. I'm curious to see how it works. (I'll check it out soon)
Can you please add some examples of how this extension can be applied in real situations? I think it would be helpful to show it to my team and get their opinions.
Although I might know the main use case for this, that's why I was interested in the first place. but because you're the author I thought would be nice if you showed what problem we're trying to solve using this. 💐

[ne4ta]

Hi @alirezanet. Sure. Here you can see some examples:

1. Without pre-initialized mapper:

var gq = new GridifyQuery()
{
    Filter = "FirstName=John",
    Page = 1,
    PageSize = 20,
    OrderBy = "Age"
};

var response = await client.SearchAsync<User>(s => s
    .Index("users")
    .From((gq.Page - 1) * gq.PageSize)
    .Size(gq.PageSize)
    .Query(gq.Filter.ToElasticsearchQuery<User>())
    .Sort(request.Sorts.ToElasticsearchSortOptions<User>()));

return response.Documents;

It will return the collection of Users.
The query that will be sent to Elasticsearch:

GET users/_search
{
  "query": {
    "term": {
      "firstName.keyword": {
        "value": "John"
      }
    }
  },
  "from": 0,
  "size": 20,
  "sort": [{
    "age": {
      "order": "asc"
    }
  }]
}

2. With custom mappings:

var gq = new GridifyQuery()
{
    Filter = "name=John, surname=Smith, age=30, totalOrderPrice=45",
    Page = 1,
    PageSize = 20,
    OrderBy = "Age"
};

var mapper = new GridifyMapper<User>()
     .AddMap("name", x => x.FirstName)
     .AddMap("surname", x => x.LastName)
     .AddMap("age", x => x.Age)
     .AddMap("totalOrderPrice", x => x.Order.TotalSum);

var response = await client.SearchAsync<User>(s => s
    .Index("users")
    .From((gq.Page - 1) * gq.PageSize)
    .Size(gq.PageSize)
    .Query(gq.Filter.ToElasticsearchQuery(mapper))
    .Sort(request.Sorts.ToElasticsearchSortOptions(mapper)));

return response.Documents;

The query to Elasticsearch:
The query that will be sent to Elasticsearch:

GET users/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "firstName.keyword": {
              "value": "John"
            }
          }
        },
        {
          "term": {
            "lastName.keyword": {
              "value": "Smith"
            }
          }
        },
        {
          "term": {
            "age": {
              "value": 30
            }
          }
        },
        {
          "term": {
            "order.totalSum": {
              "value": 45
            }
          }
        }
      ]
    }
  },
  "from": 0,
  "size": 20,
  "sort": [{
    "age": {
      "order": "asc"
    }
  }]
}

3. By default, Elasticsearch for document fields it converts property names to camel-case. That's how these two extensions work by default. But if it's necessary to apply custom naming policy, it can be also customized:

Func<string, string>? namingAction = p => $"_{p}_";
var mapper = new GridifyMapper<TestClass>(autoGenerateMappings: true)
{
   Configuration = { CustomElasticsearchNamingAction = namingAction }
};

var gq = new GridifyQuery()
{
    Filter = "FirstName=John",
    Page = 1,
    PageSize = 20,
    OrderBy = "Age"
};

var response = await client.SearchAsync<User>(s => s
    .Index("users")
    .From((gq.Page - 1) * gq.PageSize)
    .Size(gq.PageSize)
    .Query(gq.Filter.ToElasticsearchQuery(mapper))
    .Sort(request.Sorts.ToElasticsearchSortOptions(mapper)));

The query that will be sent to Elasticsearch:

GET users/_search
{
  "query": {
    "term": {
      "_FirstName_.keyword": {
        "value": "John"
      }
    }
  },
  "from": 0,
  "size": 20,
  "sort": [{
    "_Age_": {
      "order": "asc"
    }
  }]
}

[alirezanet]

I quickly checked your PR and overall it is awesome 🚀, but there are a few problems we need to fix before adding this.

• I noticed we have a lot of duplicate code which makes it really hard to maintain this later especially if users start using it, instead of coping the existing code would be better if you could make the internal APIs more flexible and reusable in a way to support your extensions.
• You didn't add any of these features to the QueryBuilder class which is the second way of creating queries in Gridify,
• I saw in your example there is a manual paging logic, would be nice if you could add the ApplyPagination support to ElasticExtensions or even Gridify() to do everything at once

 .From((gq.Page - 1) * gq.PageSize)
 .Size(gq.PageSize)

let me know what you think and even if you disagree with my comments, so we can talk about it openly. and again thank you for your contribution. 💐

[alirezanet]

Parsing decimal and double values is a Culture-specific operation, for example, on my machine, 18 tests failed because my decimal separator is a ,, this is not a solution to fix this problem in Gridify, but for now add this to your tests to fix it in everyone's machines. (later we need to add CultureInfo support to gridify)

GridifyGlobalConfiguration.CustomElasticsearchNamingAction = p => p;
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture; // <--

[alirezanet]

There are some warnings that if you already have null checks like this example, it is easy to fix using a ! operator.

var syntaxTree = SyntaxTree.Parse(filter!, GridifyGlobalConfiguration.CustomOperators.Operators);

Github analyzer already marked these so I'll skip other ones.

[alirezanet]

No Action is needed.
for now, it is okay to have this setting here but ideally would be better if we find another way to keep other extensions's configurations separated. (Just said this in case you could find a cleaner solution for this problem)

[alirezanet]

same as global configuration

[alirezanet]

Try to avoid duplicate code.

[alirezanet]

I should note, I know how complicated this convertors are and also internal Gridify expression builder is not easy to read, so let me know if I can help on this or you had any questions. maybe this is a good reason also for me to clean up the internal APIs

[ne4ta]

Hi @alirezanet. Thank you for your review. I will try to improve the code as much as I can.

[ne4ta]

Hi @alirezanet. I've updated the PR with all necessary changes. I also updated the documentation. Furthermore, I had to exclude tests from formatting in husky settings cause it broke my tests.

Also, I wanted to ask you about versioning of Gridify.Elasticsearch. It uses Elastic.Clients.Elasticsearch 8.* and this is the lowest version of this lib. Previously, the NEST lib was the official Elasticsearch client for .NET. And the version of the official client corresponds to the version of Elasticsearch that is supported. Do you think it makes sense to start Gridify.Elasticsearch with 8th version as well? E.g. Gridify.Elasticsearch 8.2.11.1 where 8 is a version of Elasticsearch and 2.11.1 is a version of Gridify.

Or maybe you have other thoughts about the versioning.

[alirezanet]

Hi @ne4ta,

Versioning:

• EF Core also follows a similar versioning pattern, but I chose to ignore it and instead relied on a range of supported versions as dependencies. We can adopt a similar approach for Elasticsearch, maintaining versioning internally within the library. This should work smoothly unless they introduce any breaking changes to the main API. In such a case, we can release a new version.

• Furthermore, since Gridify.Elasticsearch utilizes different APIs and extensions, I believe it's better to implement separate versioning, such as 0.0.1.

• But honestly I don't have any strong opinion on this.

Documentation:

• There have been significant improvements in the documentation, but I'd like to request that you maintain the Gridify.Elasticsearch details on a dedicated page. For instance, the extension methods we support could have their own page or a menu with subpages. The inclusion of multiple warnings on the base documentation pages and numerous examples, while valuable, might lead to user confusion.

• Therefore, please retain the original documents as they were, but feel free to introduce new menus/pages for Elasticsearch-related documentation.

Code:

The code looks improved substantially, but I plan to review and test it shortly. I'll inform you if any changes are necessary.

Question

Would it be acceptable for me to commit changes to your branch if I come across any easily fixable issues during the review in your branch?

Ultimately, I want to express my gratitude for your tremendous effort and innovative idea. I believe that this extension will become highly popular and beneficial for Gridify in the near future.

[ne4ta]

Hey @alirezanet,

Versioning:

Yes, I agree with the same approach that is used in Gridify now. Let's have the same approach for all packages.

Documentation:

Sure, I will redo the doc. That can really be very confusing for the users.

Question:

Please feel free to do any changes in my PR if it does not require a discussion.

And thanks for your kind words! I'm looking forward to the first release of the lib :)

[ne4ta]

Hi @alirezanet. I've updated the documentation. I hope now it looks better.

[alirezanet]

Hi @ne4ta,
The new documentation is much better; I just think you need to remove the QueryBuilder side menu from the elasticsearch section since it is irrelevant.

---

And about the extension methods, I think adding an extension method to a string is too general and maybe not the best choice. (ToElasticsearchQuery and ToElasticsearchSortOptions )
According to your example, this is how people are going to use it, right?

var query = "name = John".ToElasticsearchQuery<User>();

await client.SearchAsync<User>(s => s
    .Index("users")
    .Query(query));

Since this SDK doesn't support strings, why don't you directly add support for string queries?

await client.SearchAsync<User>(s => s
    .Index("users")
    .Query("name = John"));

This way, we're adding another extension overload to this method.
something like this maybe:

  public static SearchRequestDescriptor<T> Query<T>(this SearchRequestDescriptor<T> searchRequestDescriptor, string filter, IGridifyMapper<T>? mapper = null)
   {
      var query = ToElasticsearchQuery(filter, mapper); // we can make ToElasticsearchQuery internal or private
      searchRequestDescriptor.Query(query);
      return searchRequestDescriptor;
   }

EDIT: but on the other side this is the same as ApplyFiltering. 🥴so nvm.

for now if you want to keep these two extensions, they should be added to the IGridifyFiltering and IGridifyOrdering interfaces.

Let me know what you think.

[alirezanet]

Awesome 💐🙌,

Everything looks good, I'll add some small changes to fix the Husky and formatter problems then I'll merge it 😎

[alirezanet]

Hi @ne4ta,
Your extension is now publicly available https://www.nuget.org/packages/Gridify.Elasticsearch 🚀🍻

FYI: Just to keep the versioning simple and avoid any confusion, I started with the same version as the Gridify base project.

[ne4ta]

Hi @alirezanet,

Great news! Thanks for your help!

[alirezanet]

Hi @ne4ta,
I'm planning to review the absent feature in the Elasticsearch package (issue #132). However, before I proceed, I'd like to gather your thoughts. Was there a specific reason this feature wasn't part of the initial release that I may not know about?

[ne4ta]

Hi @alirezanet. It looks like I didn't fully understand the concept of filtering using collections. I saw in the doc that you can use it like this prop[8] > 10. So this feature can be used only for LINQ. It's impossible to translate it into an SQL query. The same applies to Elasticsearch; you cannot filter by using a specific index of an array.
However, I overlooked that there are valid cases for filtering with collections without specifying an index. E.g.:

• if nested type is used

GET /music/_search
{
  "query": {
    "nested": {
      "path": "Albums",
      "query": {
        "bool": {
          "must": [
            { "match": { "Albums.Name": "Meteora" } }
          ]
        }
      }
    }
  }
}

• for flattened objects

GET /music/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "Albums.Name": "Meteora" } }
      ]
    }
  }
}

We definitely need to add the capability to search in such a way. I will implement this feature. I can't say how much time will it take, but I will do it when I have free time.

[alirezanet]

Hi @ne4ta,
thank you for looking into this, yes we don't need an indexing feature but searching in subcollections we probably can support.
here is a small documentation: https://alirezanet.github.io/Gridify/guide/gridifyMapper.html#filtering-on-nested-collections

let me know if I can help, also you can contact me on Discord if you have any questions.