New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make lifter able to query any kind of data (LINQ) #33

Closed
EliotBerriot opened this Issue Mar 6, 2016 · 1 comment

Comments

Projects
None yet
1 participant
@EliotBerriot
Owner

EliotBerriot commented Mar 6, 2016

At the moment, lifter is focused on querying against Python iterables.

However, there is an opportunity to make it much more than that. Some people, (e.g. on reddit) mentioned Microsoft .NET LINQ as a possible use case for lifter, and I recently stumbled accross pynq, an attempt to implement this in Python.

As you can see in the project wiki, there are indeed some similitude between our API, but LINQ (and therefore, pyqn) goals are much wider than lifter at the moment.

Basically, LINQ (Language Integrated Query) is a way to query any kind of provider, using the same API everytime. A provider can be a collection/python iterable, a REST API, a relational database or basically anything that can return results.

Of course, where not Microsoft, and we're building a package here, not a Python implementation, so we'll probably never reach this kind of clean syntax:

var results =  from u in users
               where u.age < 10 AND u.is_active = True
               select new (u.first_name, u.email);

But imagine if you could query an IMAP mailbox with lifter:

from lifter.backends.imap import ImapBackend, Email

backend = ImapBackend(host='imap.example.com', user='test', password='test')

# Get unread email from INBOX
backend.select(Email).filter(Email.unseen == True, Email.directory == 'Inbox')

Or a REST API:

from lifter.backends.http import RESTBackend, JSONModel

class MyStoreApi(RESTBackend):
    endpoint = 'https://store.example.com/api/'
    content_type = 'json'

class Product(JSONModel):
    path = 'products'

backend = MyStoreApi()
for product in backend.select(Product).all():
    print(product.title, product.description)

Or a SQL Database:

from lifter.backends.sql import SQLiteBackend, SQLModel

class User(SQLModel):
    table_name = 'users'

SQLiteBackend('/path/to/db').select(User).filter(User.is_active == False).count()

And, of course, a Python iterable:

from lifter.backends.python import PythonModel as APIResult

manager = APIResult.load(api_results)
sorted_results = manager.order_by(~APIResult.relevance)

I can imagine an infinite number of use cases for such a set of features. If I sum it up, lifter would grow from an iterable query engine, to a generic query engine.

What does it imply:

  • All iterable-specific code should be move to a dedicated backend
  • Queries should be backend independent, and will be intepreted by backend
  • Backend results (SQL Query, HTTP JSON response, etc.) would be translated
    in plain python objects after the query is executed, that's probably a job for the Model class.

That would be pretty big change, but also an exciting one because it opens so many possibilities.

@EliotBerriot

This comment has been minimized.

Show comment
Hide comment
@EliotBerriot

EliotBerriot Jul 12, 2016

Owner

I'll close this, as the converting part is now done using adapters, see 82b0692

Owner

EliotBerriot commented Jul 12, 2016

I'll close this, as the converting part is now done using adapters, see 82b0692

EliotBerriot pushed a commit that referenced this issue Jul 12, 2016

Eliot Berriot
Merge tag '0.3' into develop
0.3 (2016-7-12)
---------------

This is a big release, that breaks backward-compatibility with previous ones.

This release implements a new flow to help implementing #33. The general idea
is to make lifter generic and be able to query any data source with it.

The 0.3 release sets the foundation for that by moving all python-iterable related code to a dedicated backend,
and by implementing the Store -> Adapter > Model layout to deal with queries and result parsing.

An additional, very simple, ``filesystem`` backend is provided to demonstrate how you can implement your own datasource in lifter.

The work, though, is still incomplete, because the `filesystem` store internally uses the `IterableStore` from the python backend.

A real store (such as REST or SQL) would be able to understand queries and pass them to a real backend (PostgreSQL).

Anyway, we're in the good direction here :)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment