Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Make lifter able to query any kind of data (LINQ) #33
At the moment, lifter is focused on querying against Python iterables.
However, there is an opportunity to make it much more than that. Some people, (e.g. on reddit) mentioned Microsoft .NET LINQ as a possible use case for lifter, and I recently stumbled accross pynq, an attempt to implement this in Python.
As you can see in the project wiki, there are indeed some similitude between our API, but LINQ (and therefore, pyqn) goals are much wider than lifter at the moment.
Basically, LINQ (Language Integrated Query) is a way to query any kind of provider, using the same API everytime. A provider can be a collection/python iterable, a REST API, a relational database or basically anything that can return results.
Of course, where not Microsoft, and we're building a package here, not a Python implementation, so we'll probably never reach this kind of clean syntax:
But imagine if you could query an IMAP mailbox with lifter:
from lifter.backends.imap import ImapBackend, Email backend = ImapBackend(host='imap.example.com', user='test', password='test') # Get unread email from INBOX backend.select(Email).filter(Email.unseen == True, Email.directory == 'Inbox')
Or a REST API:
from lifter.backends.http import RESTBackend, JSONModel class MyStoreApi(RESTBackend): endpoint = 'https://store.example.com/api/' content_type = 'json' class Product(JSONModel): path = 'products' backend = MyStoreApi() for product in backend.select(Product).all(): print(product.title, product.description)
Or a SQL Database:
from lifter.backends.sql import SQLiteBackend, SQLModel class User(SQLModel): table_name = 'users' SQLiteBackend('/path/to/db').select(User).filter(User.is_active == False).count()
And, of course, a Python iterable:
from lifter.backends.python import PythonModel as APIResult manager = APIResult.load(api_results) sorted_results = manager.order_by(~APIResult.relevance)
I can imagine an infinite number of use cases for such a set of features. If I sum it up, lifter would grow from an iterable query engine, to a generic query engine.
What does it imply:
That would be pretty big change, but also an exciting one because it opens so many possibilities.
0.3 (2016-7-12) --------------- This is a big release, that breaks backward-compatibility with previous ones. This release implements a new flow to help implementing #33. The general idea is to make lifter generic and be able to query any data source with it. The 0.3 release sets the foundation for that by moving all python-iterable related code to a dedicated backend, and by implementing the Store -> Adapter > Model layout to deal with queries and result parsing. An additional, very simple, ``filesystem`` backend is provided to demonstrate how you can implement your own datasource in lifter. The work, though, is still incomplete, because the `filesystem` store internally uses the `IterableStore` from the python backend. A real store (such as REST or SQL) would be able to understand queries and pass them to a real backend (PostgreSQL). Anyway, we're in the good direction here :)