Deprecate(or remove) undocumented features/code

[vmuriart]

There are a few functions/classes that have limited usage and aren't covered in the documentation.
Their functionality (ex. stripwhitespace) overlaps with similar coded filters (whitespace_filter in this case).
I did a quick look up for any usage of them and only found them being used on antiORM.

The complete commit for removal can be seen here vmuriart@836f8d8

@andialbrecht @piranna Any heart-burn over removing them? It greatly simplifies the code.

Waiting on my other pr before submitting this one. But figured I can get the discussion going beforehand

[andialbrecht]

I'm fine with removing stripwhitespace as the functionallity is covered by wihtespace_filter. Since the next release will be 0.2.0 removing obsolete functions sounds good.

[idank]

ColumnsSelect was also removed. How does one do that now that it's gone?

[vmuriart]

I been debating re-adding some of the functionalities. The restructure lends itself to be easier to integrate other functionality. The reason I wanted to remove it (atleast temporarily) is that it was tying up alot of the underlying framework from being re-written/optimized.

None of what i said above answers your question, but just wanted to give some background 😃
To answer your question, let me take a look at the original columnselect code and see about either reintegrating it, or provide an alternate solution.

[idank]

It's a bit unfortunate that the real parsing, e.g. everything that's going on in grouping.py and things like finding the column names is done by hand in sqlparse. It's a lot of work and ends up with clunky code like this deleted function.

I wonder how much work it would be to take this file, translate it to PLY and create a tiny library out of it that returns better data structures for traversing and extracting information out of queries.

[vmuriart]

I been looking into it, I liked grako for its (implied) speed and parsimonious for its simpler implementation of peg.
But both are way outside of my field of knowledge and was struggling to figureout where to start.

In the long road, I think it would be the better solution and would free-up sqlparse to do some of the more advance stuff.

[andialbrecht]

@idank the problem with grammar-based approaches is that they often raise errors if the parsed statement doesn't match the grammar. sqlparse has some use cases where a statement is syntactically incorrect, but it should be formatted as good as possible e.g. for debugging.

[idank]

That's fair. I guess I want a parser that returns a detailed AST more than a formatter. The current approach doesn't make it easy to extract information.

[vmuriart]

@idank thats something that hopefully will improve.

As a proof-of-concept though I polished my regex skills (which sqlparse has helped improve) and translated the sql-92 document to ebnf and fed it to grako

I haven't been able to get it to parse select 1 from dual where 1 = 1. It almost feels like a step backwards; similar to depending on pygments to perform the lexing/parsing for us. @andialbrecht is it safe to assume that you want sqlparse to have 0 dependencies outside the standard library?

I uploaded the code and references in case someone else wants to continue working on it; or wants to find the ebnf definition for sql.

[idank]

I honestly think taking this file and associated strutures and doing a monkey level translation to Python's PLY is a couple hours work. I have a transatlantic flight in a few weeks, I might give it a go.