New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Access Layer #161

Open
jeevatkm opened this Issue Apr 7, 2018 · 9 comments

Comments

4 participants
@jeevatkm
Member

jeevatkm commented Apr 7, 2018

aah's goal of Data Access Layer as follows-

  • Support RDBMS and NoSQL data sources
  • Seamless usage of multiple data sources (be it RDMBS or NoSQL) and having one as default data source
  • Data source aware model/entity, so that aah user doesn't have to deal with hard part targeting different data source.

I will do my analysis first then planned out implementation part.


Data Access scope is vast, I'm gonna go by step by step

  • Create foundation
  • Mapping/Binding Result Sets into struct
  • CURD style operations
  • Persistence events/hooks
  • Multiple Database support (to begin with MySQL & Postgres)
  • Multiple DataSource support and connection management
  • Prepared Statements with Named Arguments binding from struct, map
  • Native query execution
  • Query Cache in-memory to begin with

Upcoming List

  • Database tables to struct Model generation type safe
  • Datbase Migration (up & down) with audit-trail
  • Transactions
  • Data Source aware models
  • Debug logging (first release would have basic level)
  • Filters/Processors
  • etc

@jeevatkm jeevatkm created this issue from a note in aah Roadmap (Backlog) Apr 7, 2018

@jeevatkm jeevatkm moved this from Backlog to v0.11.0 - Iteration in aah Roadmap Apr 7, 2018

@jeevatkm jeevatkm added this to the v0.11.0 Milestone milestone Apr 7, 2018

@jeevatkm jeevatkm added the feature label Apr 8, 2018

@broklyngagah

This comment has been minimized.

broklyngagah commented Apr 10, 2018

I use gorm in all of my RDBMS projects. +1 for gorm..

@jeevatkm

This comment has been minimized.

Member

jeevatkm commented Apr 10, 2018

@broklyngagah Thank you for your inputs.

I'm currently in the process of figuring out and provide meaningful design and usage - As you know
"Modern application uses RDBMS as well as NoSQL".

@blagodus

This comment has been minimized.

blagodus commented Apr 12, 2018

@jeevatkm you can look at these packages for inspiration(integration):

Do you plan to build new package or just integrate existing?

@jeevatkm jeevatkm changed the title from Database Access Layer to Data Access Layer Apr 12, 2018

@jeevatkm

This comment has been minimized.

Member

jeevatkm commented Apr 12, 2018

@blagodus Thank you for your inputs, I will surely look into it. It seems in the initial write up my goals didn't come out properly. I have just update it.

Answer to your question, I have not yet decided the approach whether new package or integrating existing. I will be doing my analysis inline with goals of this implementation.

Once analysis is done, I would publish my direction on this thread.

@AugustHell

This comment has been minimized.

AugustHell commented Apr 25, 2018

Seamless usage of multiple data sources (be it RDMBS or NoSQL) and having one as default data source

You might have a look on the awesome CockroachDB. It is written in go, incredible fast, made for scaleability and it can be connected thru postgres-adapters. JSON types makes it interesting for NoSQL users too.

@jeevatkm

This comment has been minimized.

Member

jeevatkm commented Apr 25, 2018

@AugustHell Thank you for your inputs.

@AugustHell

This comment has been minimized.

AugustHell commented Apr 26, 2018

The database layer is a big topic in my eyes. Most frameworks struggle on that part or even touch it.
In the go world I haven't find yet a really good one.
Some thoughts:

Multiple Connection Handling

As write operations are the heavy impacts for any database, a solution for high traffic sites is to have a pure read and a read/write connection to two different servers that are handling the sync on their own.
Another real life example are geo located connections to serve data from the nearest db server to keep latency low.

Permanent and Temporary Connections

The number of db connections is most time limited, so a permanent connection seems great, but is not in any case. A developer should have the freedom of choice to use what he needs.

Database Modeling

I really like to have a structured model for each db table I use. In projects you need a lot of tables, you get quick bored creating them. The helping methods I know of, are creating thru command line scripts, auto migration like in gorm, or extracting from the database's "SHOW TABLE".
While some databases have great modeling tools, like eg MySQL Workbench, others don't. Hence offering only one way of modeling is limiting it's use.

Topic based Migrations

When it comes to migrations the most seen solution is a time based one with up/down scripts. In my opinion, this is not really practicable as when developing in a team on a bigger project not every one is focused on the same tables, Soon there are a lot of up and downs you have to do if you want to revert one manipulation. A table based versioning reflecting the table relations would be nicer.

Multiple Query Cache

While most databases have a query cache, table joins are sometimes not the most perfomant solution. So you hack your data needs with serveral querys together. Putting that in a cache can speed up the app immense. So if it comes to a query builder, some kind of transaction block which aim is not the data integrity but the caching of the output would be nice to have.

That's not a complete list, just what came to my mind at the moment.

@jeevatkm

This comment has been minimized.

Member

jeevatkm commented Jun 8, 2018

Update: My homework and analysis are progressing well on data access layer, however I need further time to start foundation. So pulling this one out from v0.11.0 and making v0.11.0 Milestone release.

Thank you very much for the support and understanding.

@jeevatkm jeevatkm removed this from the v0.11.0 Milestone milestone Jun 8, 2018

@jeevatkm jeevatkm moved this from v0.11.0 - Iteration to Backlog in aah Roadmap Jun 9, 2018

@jeevatkm jeevatkm self-assigned this Jul 14, 2018

@jeevatkm

This comment has been minimized.

Member

jeevatkm commented Jul 14, 2018

I have updated with plan and pulled out from initial and adding it here for reference:


Following is initial leg work on RDBMS findings-

I have done my homework (read documentation, source code) around many data access layer libraries exists now. It turns out following is the viable candidates for integration as data access layer with aah.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment