Elastic Search Connector

[sorenbs]

Prisma supports two kinds of database connectors:

• Active connectors - manage and migrate the database schema
• Passive connectors - read and write data only

This is a living document outlining all considerations for an active SQL connector. This is not a final spec - please contribute by commenting below.

Simple query

{
  Match: {
    last_name: "Smith"
  }
}

Combined filter and query

{
  filtered: {
    filter: {
      range: {
        age: { gt: 30 }
      }
    }
    query: {
      match: {
        last_name: "Smith"
      }
    }
  }
}

returns

{
  ...
  "hits": {
    "total": 1,
    "max_score": 0.30685282,
    "hits": [
      {
        ...
        "_source": {
          "first_name": "Jane",
          "last_name": "Smith",
          "age": 32,
          "about": "I like to collect rock albums",
          "interests": [ "music" ]
        }
      }
    ]
  }
}

Questions

• Should we keep the _ prefixes used by elastic search in many places?

[siddharthlatest]

Love that you are thinking about the Elasticsearch adaptor.

I am not sure how this would map to the full Elasticsearch Query DSL? Even if you narrowly consider match, term and bool as three clauses to support - there are numerous parameters each of them can support. Can you share how an example spec might look like?

Also, if the idea isn't to fully support the DSL (imo really hard to do), what is the fallback mechanism?

[sorenbs]

@siddharthlatest With all connectors we aim to support 90+ % of the underlying functionality. The reason we don't aim for 100% is that adding the last bit of functionality would add significant clutter to the API degrading the developer experience. We will support arbitrary queries defined as a string in the data model as fallback. While this is more tedious, it enables the full power of Elastic while keeping most common functionality easily accessible. We will expand on the spec in the next couple weeks and look forward to your feedback :-)

[siddharthlatest]

@sorenbs Looking forward to the expanded spec. In the meanwhile, is there any example spec (may be for a different DB) that can give an idea on how this might work?

[nimi]

Any updates on the spec and/or progress on this feature?

[thebetterjort]

Does anybody have an example?

[dan-kez]

Hi all! Just wanted to check in and see if this is still on the 2018-Q4 roadmap as we're about to enter Q4

[Disturbing]

Yes, looking forward to this - is it still on track?

[mrimran]

Looking forward for this feature. Please update on this.

[siddharthlatest]

I am interested in contributing and/or collaborating on this in some way - do we have any initial plans / spec?

[sorenbs]

Thank you all for the interest in this! We have not yet started work on the Elastic connector, and we will not be able to start this year. I will update this thread when we have a clear roadmap.

If you are interested in the general concept on non-sql database connectors you can already check out the Mongo Preview

We would like critical feedback from the community on the proposed API spec above. If anybody would be willing to jump in and create a complete specification, that would also be tremendously helpful. Feel free to open a separate issue and link it here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

[ali-habibzadeh]

Will this still happen?

[dzcpy]

Any updates?

[Disturbing]

Anyone want to take this up on a bounty? KintoHub will pay for it :)

[dzcpy]

@Disturbing I might give a try, how do I contact you?

[Disturbing]

To refrain from putting my email address up - please join our kintohub slack - reach out to me directly (Same alias as here)

[janat08]

why not prioritize arangodb which has arangosearch as competitor to elasticsearch.

[suminda123]

any updates

[tvvignesh]

@sorenbs Hope this is still under consideration for prisma2 as well. I am using Postgres & ELK combination in my stack with ELK being used for time series data. Just wanted to make sure since this issue is filed under Prisma1 repo.

[bionicles]

+1

[lucasbastianik]

+1