feat: Native PG array type

[etimberg]

No description provided.

[maitham]

Can I help at all in getting this merged?

[etimberg]

@maitham I think the big thing this needs now is to figure out how to get contains and contained_by queries working. I briefly got some tests in but they are currently failing. I will push those up now and we can go from there

[etimberg]

I made some progress on debugging these tests and trying to make this work. Basically it boils down to needing to call .contains() on the column and then passing in the correct parameters.

The challenge is that when Ormar compiles statements, it uses literal_binds=True. https://github.com/collerek/ormar/blob/master/ormar/queryset/actions/filter_action.py#L192

Unfortunately, SQLAlchemy cannot render literals for arrays. A simple test to show this is:

from sqlalchemy import Column, Integer
from sqlalchemy.dialects import postgresql

numbers = Column(postgresql.ARRAY(Integer))
clause = numbers.contains([11])
print(clause.compile(compile_kwargs={"literal_binds": True},))  # Causes a NotImplementedError

@collerek is there a reason ormar uses literal_binds=True? Trying to generate strings for PG arrays seems like it will be complicated and error prone.

[etimberg]

Following up on literal_binds=True, SQLAlchemy does not recommend this

[collerek]

Following up on literal_binds=True, SQLAlchemy does not recommend this

Will check that, to be honest I don't remember why it uses literal binds 🙈

[etimberg]

Thanks for taking a look @collerek! I was digging through it as well. It looks like it was introduced in collerek/ormar#87. It appears to be related to column aliases. I was actually able to make it work with arrays by simply commenting out the call to _compile_clause in filter_action.py but I suspect that breaks a lot of things elsewhere (I haven't tried the ormar test suite with this change).

The good news is that I haven't been able to inject something via a string filter. sqlalchemy.text may be preventing a lot of things.

[collerek]

There is a PR that removes the literal binds, currently blocked by a bug in encode\databases that was patched but not yet released. The bug refers only to sqlite so you can probably check with the PR branch as should already work for postgres.

[etimberg]

Amazing, thank you @collerek!

[etimberg]

@collerek not sure how you feel about this patching of the FieldAccessor class and the FILTER_OPERATIONS and METHODS_TO_OPERATORS dictionaries. I'm open to other suggestions if there's a better way to implement this.

[tophat-opensource-bot]

🎉 This PR is included in version 2.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀