Fixed #18757, #14462, #21565 -- Reworked database-python type conversions

[mjtamlyn]

Make resolve_columns a permanent fixture, remove convert_values.

Current implementation passes on sqlite at least, but does destroy performance. Need a nicer converter-collection based approach. Also some tests that we can use the new override from_db_value, and that #21565 is fixed in the process.

• Make the tests pass
• Refactor to use "convertor-collector" based system
• Add tests for #21565
• Add tests for #14462
• Add documentation for from_db_value and appropriate test suite for it

[mjtamlyn]

Woo! We have a green light. http://djangoci.com/job/django-pull-requests/738/

Updating the description with the current todo list.

[timgraham]

Field.from_db_value()

[shaib]

append(...), not append[...].

[mjtamlyn]

Status update:

I've now got djangobench reporting no more than a 1.1x slowdown on any benchmark, with most running the same. There is basically no change in the performance on any standard benchmark. There will be a performance hit when resolve_columns is run, but on mysql at least this will now happen less often than it used to so a net win.

Documentation has been all tidied up as per @timgraham's comments.

I have some issues with some other databases which the CI server is finding which I should hopefully be able to pin down tomorrow morning. Once these are resolved we should be good to merge.

Any comments @akaariai?

[mjtamlyn]

The CI is very flaky this morning... The most recent test run has a few configurations which died out due to failed checkouts, and some of the Oracle builds refused connections.

The one oracle build which worked - http://djangoci.com/job/django-pull-requests/814/database=oracle11,python=python2.7/console - seems to have a bunch of errors which I know I fixed in 86c2404, as well as a strange collection of unrelated failures largely seeming to result from a language leak. I guess this might happen where some other test is failing.

I'm going to try to retrigger the build and see if it is any happier second time round.

[timgraham]

comma after aggregation
"it has" on next line

[shaib]

Great work, overall. The amount of clean-up is impressive.

One optimization proposal: it seems that connection.get_db_converters(self, internal_type) would benefit from memoizing, or even explicitly building a dictionary internal_type->converter_list for all the core internal_types.

[shaib]

SELECT * FROM (SELECT "_SUB".*, ROWNUM AS "_RN" FROM (%s) "_SUB" %s) WHERE ...

(ROWNUM AS "_RN" should be part of the SELECT clause, not FROM clause).