Transactional key value api

[YourTechBud]

Jira Link: DB-4713
Can yugabyte expose a transactional key value api similar to what foundation db has?

[rkarthik007]

Hi @noorainp,

We have this in our longer term roadmap, unfortunately no short term plans for this since there is not enough demand. We have optimized the YCQL (Cassandra) codepath for performance so much, that that would be a good way to achieve this goal natively. Could I ask you for some more details so we can prioritize our roadmap accordingly?

[YourTechBud]

So I'm taking inspiration from foundationdb and TiKV here. The idea that you can create your own layers to extend the database just makes the same core more usable.

Also, a lot of my code is based on MongoDB, adding a MongoDB layer to extend Yugabyte would help me port to it immediately. Unfortunately the JSON support provided currently (in YCQL) won't work for me.

Yugabyte seems to have a solid architecture. It would be great if it's core could be exposed as well. This could open up a whole new dimension of community contributions as well.

[schoudhury]

@noorainp excellent point. as @rkarthik007 noted, at some point we can think of exposing a lower-level API directly. unfortunately, the current project does not have that as a goal. our recent post highlights some issues in the FoundationDB like approach https://blog.yugabyte.com/7-issues-to-consider-when-evaluating-foundationdb/ -- by directly providing 3 APIs/models we aim to avoid those.

regarding your comment on YCQL JSON, would be great if you can highlight what exactly you see as the missing features. we have been adding depth to this particular area at a steady pace so aligning them to your needs would be a lot easier.

[YourTechBud]

Yes. I've gone through your blog post. Its really cool. If I understand it correctly, it speaks more about the problems with Foundationdb rather than the "layers" approach. I totally agree with the points you've mentioned. Infact I've written a post on why the layers approach works well (a debate for another day).

I understand if it's not a current priority for you guys right now. Not much which can be done there.

I've been working on this tool which provides instant rest APIs on any database out there. The query generation block I've used cannot spit out the syntax used for the json type.

I had another question for that feature though. Is it possible to create a secondary index on a field (or sub field) present in the json object. Assuming the type of the field indexed will always be the same.

[kmuthukk]

Regarding << I had another question for that feature though. Is it possible to create a secondary index on a field (or sub field) present in the json object. Assuming the type of the field indexed will always be the same. >>>

Yes - this is work in progress right now (i.e. under development). See #741

[rkarthik007]

I've been working on this tool which provides instant rest APIs on any database out there. The query generation block I've used cannot spit out the syntax used for the json type.

In YCQL, we have kept the JSONB syntax as close as possible to the Postgres syntax. Note sure if that helps.

Is it possible to create a secondary index on a field (or sub field) present in the json object. Assuming the type of the field indexed will always be the same.

Yes this is something that we are working on. You can follow along here: #741

[YourTechBud]

Great. I think this answers it. You guys can close this issue.

[kmuthukk]

YugaByte DB has a layered architecture.

The lower half of a "distributed key-to-document storage layer" is a sharded, clustered, transactional, and log-structured key-to-document storage engine.

This lower half is designed to support multiple feature rich data models (upper half). Today, it powers three upper halfs (YEDIS, YCQL & YSQL) and is designed to be extensible for adding additional data models with ease.

Although another new data model is not on the team's current roadmap, if there's community interest for contributing to building new data models (such as MongoDB style) on top of the common core, they are totally welcome. Would love to work with folks on that.

[YourTechBud]

I'd love to contribute in any way possible (tutorials, web services, blogs, etc.). I personally believe in Yugabyte's Architecture and am a KV api away from becoming a die hard fan.

I'm planning to make a mongo layer using TiKV in golang. I need a scalable mongodb. Im sure many do. MongoDB has a huge community. Im sure most hate it when comes to deploying MongoDB on the cloud.

[rkarthik007]

That is very interesting @noorainp about building a MongoDB layer. One question - do you prefer to build it in golang? Is C++ an option at all? Asking because we can try to explore the API for the lower (which is already a document database), but choice of language may have an impact. Of course, this is very exploratory for us also.

[YourTechBud]

I'm afraid I haven't played around with c++ much. The last project i worked on was trying to build a marathon rip off for apache mesos years back (hadn't heard of nomad back then).

I feel I'm pretty good with golang though. Golang attracts much larger communities for such architecturally heavy projects since it gives us a great concurrency model to play around with. Abstracts away a lot of things. C++ forces you to play with the nitty gritties.

You could consider a microservice based extension to yugabyte instead of building everything into the same codebase. Much easier to navigate. Users can run stateless "layer" services (not necessarily as a sidecar) which could connect to the yugabyte core. Using something like Cassandra's token aware policy will make this pretty efficient as well. All we need to do is expose the core APIs (gRPC comes handy here).

But I wouldn't mind brushing up on my c++ skills for something exciting.

I'd love to hear how you guys are planning to go about it though.

[rkarthik007]

Got it... extension to the codebase makes complete sense, that may be fine. I was asking if C++ works because of the following points:

• Its a lot easier for us to support this in C++ since YCQL and YSQL both use the same set of C++ APIs for exercising the underlying transactional properties. It might be easier to write document how to write code that "uses" these libraries. However, If that needs to be in a different language, then we need to expose these as RPC service APIs, which makes more work (at least based on the items in the roadmap at present).

• The underlying API performs a lot of optimizations such as short-circuit the RPC layer and make a direct function call if the client and server are on the same machine (the case of a smart client), which in turn makes it very high in performance. These would get converted into RPCs if the client is in a different language.

[YourTechBud]

Ah. I totally missed the smart client bit. Pretty cool. I agree that for the near future, additing stuff to the existing codebase makes sense.

Would love to contribute in any way possible.

[ddorian]

@YourTechBud are you still interested ?
We have a contributors guide https://docs.yugabyte.com/latest/contribute/ and a good set of starter issues https://github.com/YugaByte/yugabyte-db/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22

[YourTechBud]

I'll definitely have a look to see if I can make myself useful in anyway.