Fixed #28574 -- Add Queryset.explain() method #9053
Conversation
ca66689
to
9320d84
@@ -25,6 +25,7 @@ class DatabaseOperations(BaseDatabaseOperations): | |||
'PositiveSmallIntegerField': 'unsigned integer', | |||
} | |||
cast_char_field_without_max_length = 'char' | |||
explain_prefix = 'EXPLAIN' |
jschneier
Sep 10, 2017
•
Contributor
Need a trailing space here I think.
Need a trailing space here I think.
orf
Sep 10, 2017
Author
Contributor
I don't think so, it's used in the as_sql
function which is a list of tokens, joined with a space
I don't think so, it's used in the as_sql
function which is a list of tokens, joined with a space
for verbose, format in itertools.product((True, False), all_formats): | ||
with self.subTest(verbose=verbose, format=format): | ||
r = qs.explain(format=format, verbose=verbose) | ||
self.assertTrue(isinstance(r, str)) |
atombrella
Sep 11, 2017
•
Contributor
Use self.assertIsInstance
instead.
Use self.assertIsInstance
instead.
orf
Sep 11, 2017
Author
Contributor
Done, thanks!
Done, thanks!
a6515be
to
f96c06b
68eaf7f
to
b1a7f8d
19c4b3a
to
93681bc
|
||
def test_explain_unknown_format(self): | ||
qs = Tag.objects.filter(name='test').all() | ||
with self.assertRaises(ValueError): |
atombrella
Oct 6, 2017
Contributor
Please test the value of the message. Since it's variable, depending on the backend, I think it's fine to use supported_explain_formats
with a check if it has a value.
Please test the value of the message. Since it's variable, depending on the backend, I think it's fine to use supported_explain_formats
with a check if it has a value.
orf
Oct 6, 2017
Author
Contributor
Can you elaborate? I'm not sure what you mean, how would I use supported_explain_formats
in this context?
Can you elaborate? I'm not sure what you mean, how would I use supported_explain_formats
in this context?
Tag.objects.filter(name='test').annotate(Count('children')), | ||
Tag.objects.filter(name='test').values_list('name'), | ||
Tag.objects.order_by().union(Tag.objects.order_by().filter(name='test')), | ||
Tag.objects.all().select_for_update().filter(name='test') |
atombrella
Oct 6, 2017
Contributor
Please add a trailing comma.
Please add a trailing comma.
queries. The output is different for each database and version, do | ||
not attempt to parse the output. | ||
|
||
|
atombrella
Oct 6, 2017
Contributor
Delete the extra space.
Delete the extra space.
# tuples with integers and strings. Flatten them out into strings. | ||
for row in result[0]: | ||
if not isinstance(row, str): | ||
yield " ".join(str(c) for c in row) |
atombrella
Oct 6, 2017
•
Contributor
Single-quotes. Please check other places. Maybe you could also merge the checks to a single line?
Single-quotes. Please check other places. Maybe you could also merge the checks to a single line?
@@ -46,6 +46,9 @@ class BaseDatabaseOperations: | |||
UNBOUNDED_FOLLOWING = 'UNBOUNDED ' + FOLLOWING | |||
CURRENT_ROW = 'CURRENT ROW' | |||
|
|||
# Prefix for EXPLAIN queries. None if the backend does not support this. | |||
explain_prefix = None |
atombrella
Oct 6, 2017
Contributor
Blank string instead?
Blank string instead?
|
||
def explain_query_prefix(self, output_format=None, verbose=False): | ||
if not self.explain_prefix: | ||
raise NotImplementedError("This backend does not support explaining query execution") |
atombrella
Oct 6, 2017
Contributor
NotSupportedError
, and please use single-quotes, and add a dot at the end.
NotSupportedError
, and please use single-quotes, and add a dot at the end.
@@ -48,6 +48,13 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
""" | |||
|
|||
@cached_property | |||
def supported_explain_formats(self): | |||
if self.connection.mysql_version >= (5, 6): |
atombrella
Oct 6, 2017
Contributor
You can skip this check, as MySQL 5.5 is not supported anymore, and maybe it's better to move into a class-variable.
You can skip this check, as MySQL 5.5 is not supported anymore, and maybe it's better to move into a class-variable.
@@ -51,6 +51,8 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
supports_over_clause = True | |||
supports_aggregate_filter_clause = True | |||
|
|||
supported_explain_formats = {'XML', 'JSON', 'YAML'} |
atombrella
Oct 6, 2017
Contributor
Although it's default, TEXT
is also a supported format.
Although it's default, TEXT
is also a supported format.
|
||
.. warning:: | ||
|
||
This method is only useful as an aide for debugging poorly performing |
atombrella
Oct 6, 2017
Contributor
aide
-> aid
? I'm not a native speaker, but isn't there a difference between these two?
aide
-> aid
? I'm not a native speaker, but isn't there a difference between these two?
orf
Oct 6, 2017
Author
Contributor
It is correct as is, but I could reword it to is only useful to aid debugging
if that reads better for non-native speakers?
It is correct as is, but I could reword it to is only useful to aid debugging
if that reads better for non-native speakers?
atombrella
Oct 7, 2017
Contributor
I hadn't seen the spelling "aide" before. I'm indifferent to both suggestions.
I hadn't seen the spelling "aide" before. I'm indifferent to both suggestions.
|
||
``explain()`` | ||
|
||
.. method:: explain(verbose=False, format=None) |
atombrella
Oct 6, 2017
•
Contributor
In PostgreSQL, there's also COST
and BUFFERS
. I didn't check Oracle or MySQL/MariaDB/SQLite documentation, but maybe **extra
could deal with additional backend specific features? I'm suggesting it here, as a small note in case of no changes in the documentation will address it with a note.
Please add a .. versionadded:: 2.1
.
In PostgreSQL, there's also COST
and BUFFERS
. I didn't check Oracle or MySQL/MariaDB/SQLite documentation, but maybe **extra
could deal with additional backend specific features? I'm suggesting it here, as a small note in case of no changes in the documentation will address it with a note.
Please add a .. versionadded:: 2.1
.
Thanks for the review @atombrella, I've made the requested changes I wasn't sure about adding |
db3ed35
to
c3a65f4
A few pedantic comments. |
result = ['SELECT'] | ||
result = [] | ||
if self.query.explain_query: | ||
result.append(self.connection.ops.explain_query_prefix(self.query.explain_format, |
atombrella
Oct 28, 2017
Contributor
No hanging indent.
No hanging indent.
orf
Nov 27, 2017
Author
Contributor
Fixed 👍
Fixed
self.assertIn('does not exist is not a recognised format', str(exc.exception)) | ||
|
||
if connection.features.supported_explain_formats: | ||
self.assertIn(', '.join(connection.features.supported_explain_formats), str(exc.exception)) |
atombrella
Oct 28, 2017
Contributor
Just for completeness, shouldn't Allowed formats:
also be in this assertion?
Just for completeness, shouldn't Allowed formats:
also be in this assertion?
@@ -51,6 +51,8 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
supports_over_clause = True | |||
supports_aggregate_filter_clause = True | |||
|
|||
supported_explain_formats = {'TEXT', 'XML', 'JSON', 'YAML'} |
atombrella
Oct 28, 2017
Contributor
I think the style is not to have a blank line between the feature-flags, except in the base-backend. Please also see the MySQL feature flag.
I think the style is not to have a blank line between the feature-flags, except in the base-backend. Please also see the MySQL feature flag.
e4c62fe
to
173a16b
Thanks for tackling this - it'll be useful not having to dredge up the query and drop out to a database shell. I have made many comments regarding the various options that can be provided and I think that the documentation would benefit from highlighting some of the backend-specific quirks. |
@@ -46,6 +46,7 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
SET V_I = P_I; | |||
END; | |||
""" | |||
supported_explain_formats = {'JSON'} |
pope1ni
Oct 30, 2017
Member
Perhaps worth adding a comment to explain the absence of TEXT
with respect to #9053 (comment).
Even better, TEXT
could be added and translated to TRADITIONAL
in explain_query_prefix()
.
(See the last bullet point in the documentation.)
Perhaps worth adding a comment to explain the absence of TEXT
with respect to #9053 (comment).
Even better, TEXT
could be added and translated to TRADITIONAL
in explain_query_prefix()
.
(See the last bullet point in the documentation.)
orf
Nov 11, 2017
Author
Contributor
I've added TRADITIONAL
to the supported formats, and translated text
to TRADITIONAL
in the explain_query_prefix
function
I've added TRADITIONAL
to the supported formats, and translated text
to TRADITIONAL
in the explain_query_prefix
function
if output_format: | ||
supported_formats = self.connection.features.supported_explain_formats | ||
if output_format.upper() not in supported_formats: | ||
msg = '{0} is not a recognised format'.format(output_format) |
pope1ni
Oct 30, 2017
Member
Make output_format
uppercase in the exception message so that it matches those in supported_formats
?
Make output_format
uppercase in the exception message so that it matches those in supported_formats
?
orf
Nov 11, 2017
Author
Contributor
Done 👍
Done
supported_formats = self.connection.features.supported_explain_formats | ||
if output_format.upper() not in supported_formats: | ||
msg = '{0} is not a recognised format'.format(output_format) | ||
if supported_formats: |
pope1ni
Oct 30, 2017
Member
Is this condition necessary? Surely all backends will support a default TEXT
format?
(Even if it can't be explicitly provided in the query...)
Is this condition necessary? Surely all backends will support a default TEXT
format?
(Even if it can't be explicitly provided in the query...)
orf
Nov 11, 2017
Author
Contributor
I'm not sure. In mysql's case TEXT
doesn't exist, it's some ridiculous table structure that is quite hard to reason about. I'd like to keep the default as unspecified if possible, if the database doesn't support any formats other than the default we should just error without any Allowed Formats:
message IMO. Open to discussion though.
I'm not sure. In mysql's case TEXT
doesn't exist, it's some ridiculous table structure that is quite hard to reason about. I'd like to keep the default as unspecified if possible, if the database doesn't support any formats other than the default we should just error without any Allowed Formats:
message IMO. Open to discussion though.
if output_format.upper() not in supported_formats: | ||
msg = '{0} is not a recognised format'.format(output_format) | ||
if supported_formats: | ||
msg += '. Allowed formats: {0}'.format(', '.join(supported_formats)) |
pope1ni
Oct 30, 2017
Member
Move the leading period to the initial definition of msg
.
Move the leading period to the initial definition of msg
.
orf
Nov 11, 2017
Author
Contributor
Done 👍
Done
@@ -50,6 +50,7 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
$$ LANGUAGE plpgsql;""" | |||
supports_over_clause = True | |||
supports_aggregate_filter_clause = True | |||
supported_explain_formats = {'TEXT', 'XML', 'JSON', 'YAML'} |
pope1ni
Oct 30, 2017
Member
Sort these formats alphabetically.
Sort these formats alphabetically.
orf
Nov 11, 2017
Author
Contributor
Done 👍
Done
elif verbose: | ||
# EXTENDED format is deprecated in 5.7 and above. | ||
prefix += ' EXTENDED ' | ||
return prefix |
pope1ni
Oct 30, 2017
Member
EXTENDED
is removed in MySQL 8.0 as it has become part of the default output as of 5.7. This needs to be changed to only be added for MySQL < 5.7
PARTITIONS
is also missing but the same situation as EXTENDED
above also applies. It is also mutually exclusive with EXTENDED
and FORMAT
. It is probably not worth adding the capability to get the PARTITIONS
output only for 5.6
According to the documentation the JSON format already includes the extra information provided by EXTENDED
and PARTITIONS
.
I suggest changing this method to something like the following (which includes the text format option translation mentioned in another comment):
def explain_query_prefix(self, output_format=None, verbose=False):
prefix = super().explain_query_prefix(output_format, verbose)
if output_format:
if output_format.upper() == 'TEXT':
output_format = 'TRADITIONAL'
prefix += ' FORMAT=%s ' % output_format
if self.connection.mysql_version < (5, 7) and verbose and output_format is None:
# EXTENDED and FORMAT are mutually exclusive options.
# EXTENDED is deprecated (and not required) in 5.7 and removed in 8.0
prefix += ' EXTENDED '
return prefix
This makes it easier to remove the cruft when MySQL 5.6 is no longer supported.
I'm not sure whether it is worth having the default for verbose
be None
thus turning this flag tri-state. This would would allow for raising an exception when verbose
is explicitly set to False
for MySQL >= 5.7, but I'll defer to someone else's better judgement.
EXTENDED
is removed in MySQL 8.0 as it has become part of the default output as of 5.7. This needs to be changed to only be added for MySQL < 5.7
PARTITIONS
is also missing but the same situation as EXTENDED
above also applies. It is also mutually exclusive with EXTENDED
and FORMAT
. It is probably not worth adding the capability to get the PARTITIONS
output only for 5.6
According to the documentation the JSON format already includes the extra information provided by EXTENDED
and PARTITIONS
.
I suggest changing this method to something like the following (which includes the text format option translation mentioned in another comment):
def explain_query_prefix(self, output_format=None, verbose=False):
prefix = super().explain_query_prefix(output_format, verbose)
if output_format:
if output_format.upper() == 'TEXT':
output_format = 'TRADITIONAL'
prefix += ' FORMAT=%s ' % output_format
if self.connection.mysql_version < (5, 7) and verbose and output_format is None:
# EXTENDED and FORMAT are mutually exclusive options.
# EXTENDED is deprecated (and not required) in 5.7 and removed in 8.0
prefix += ' EXTENDED '
return prefix
This makes it easier to remove the cruft when MySQL 5.6 is no longer supported.
I'm not sure whether it is worth having the default for verbose
be None
thus turning this flag tri-state. This would would allow for raising an exception when verbose
is explicitly set to False
for MySQL >= 5.7, but I'll defer to someone else's better judgement.
orf
Nov 27, 2017
Author
Contributor
Thank you for this, and for the code snippet. I'll work on adding support for this.
As I understand it mysql's explain output is very different from other databases, the default output is in the form of a single tuple, and the column names are very important in the output. Currently there isn't a way to return the column names from queries, which I think makes this a little bit useless on mysql currently. Would you agree, and would you know any workarounds? I'm a bit loathe to add support for returning column names from some of the internals just for this specific case.
Thank you for this, and for the code snippet. I'll work on adding support for this.
As I understand it mysql's explain output is very different from other databases, the default output is in the form of a single tuple, and the column names are very important in the output. Currently there isn't a way to return the column names from queries, which I think makes this a little bit useless on mysql currently. Would you agree, and would you know any workarounds? I'm a bit loathe to add support for returning column names from some of the internals just for this specific case.
orf
Nov 27, 2017
Author
Contributor
Ignore that, you can get the column names, I was just being dense 👍
Ignore that, you can get the column names, I was just being dense
pope1ni
Nov 27, 2017
Member
I think it is worth just spewing it out as a complete text blob. Otherwise we would be adding specialised parsing which is not worth it for a textual output.
I think it is worth just spewing it out as a complete text blob. Otherwise we would be adding specialised parsing which is not worth it for a textual output.
pope1ni
Nov 27, 2017
Member
Also, could you add in the handling for EXTENDED
as I had in my comment above? This will ensure that Django will output the same, regardless of the version of MySQL.
Also, could you add in the handling for EXTENDED
as I had in my comment above? This will ensure that Django will output the same, regardless of the version of MySQL.
orf
Nov 27, 2017
Author
Contributor
Done 👍
Done
def explain_query_prefix(self, output_format=None, verbose=False): | ||
prefix = super().explain_query_prefix(output_format, verbose) | ||
|
||
extra = {} |
pope1ni
Oct 30, 2017
Member
Following on from #9053 (comment) which didn't seem to have a response...
PostgreSQL has the following additional options in all supported versions: BUFFERS
, COSTS
, TIMING
.
Version 10+ also supports an additional SUMMARY
option. (See documentation for more details.)
Note that some of these options are already enabled by default but can be explicitly disabled. It could be easier to just enable all of these options when verbose is set if it doesn't hugely increase the time taken to execute the statement.
Following on from #9053 (comment) which didn't seem to have a response...
PostgreSQL has the following additional options in all supported versions: BUFFERS
, COSTS
, TIMING
.
Version 10+ also supports an additional SUMMARY
option. (See documentation for more details.)
Note that some of these options are already enabled by default but can be explicitly disabled. It could be easier to just enable all of these options when verbose is set if it doesn't hugely increase the time taken to execute the statement.
orf
Oct 30, 2017
Author
Contributor
Oh, I must have forgotten to respond to that comment. I didn't want to go too deep into adding support for all the various flags, but I think that I will after reading your comments. I'll let .explain()
accept arbitrary keyword arguments and the various explain_query_prefix
functions can do what they want with them.
Oh, I must have forgotten to respond to that comment. I didn't want to go too deep into adding support for all the various flags, but I think that I will after reading your comments. I'll let .explain()
accept arbitrary keyword arguments and the various explain_query_prefix
functions can do what they want with them.
pope1ni
Oct 31, 2017
Member
Yeah. There are quite a few, and this is anything but standard across backends. The advantage is being able to access all of this information without needing to pop out to the database shell and restricting the flags may cripple the use of this feature for some.
Adding support via kwargs sounds sensible. The irritation may be determining the default state of the various flags based on the value of verbose
. So it may be easier to keep the interface simple and just "enable all the things" if verbose
is True
, where they are not costly in terms of execution time, and then provide analyze
separately in kwargs.
Yeah. There are quite a few, and this is anything but standard across backends. The advantage is being able to access all of this information without needing to pop out to the database shell and restricting the flags may cripple the use of this feature for some.
Adding support via kwargs sounds sensible. The irritation may be determining the default state of the various flags based on the value of verbose
. So it may be easier to keep the interface simple and just "enable all the things" if verbose
is True
, where they are not costly in terms of execution time, and then provide analyze
separately in kwargs.
extra['FORMAT'] = output_format | ||
if verbose: | ||
extra['VERBOSE'] = 'true' | ||
extra['ANALYZE'] = 'true' |
pope1ni
Oct 30, 2017
Member
I don't think that ANALYZE
should be set when using verbose
. This makes it execute the query which could take a long time, especially when users will think this means VERBOSE
. Perhaps a separate parameter should be used for triggering this behaviour?
I don't think that ANALYZE
should be set when using verbose
. This makes it execute the query which could take a long time, especially when users will think this means VERBOSE
. Perhaps a separate parameter should be used for triggering this behaviour?
pope1ni
Oct 30, 2017
Member
In addition, use of ANALYZE
can lead to modification of data which should be documented - even with a SELECT
query if a function is executed - see the documentation. It might make sense to highlight that a transaction should be used which can be aborted if the user wants to avoid this.
In addition, use of ANALYZE
can lead to modification of data which should be documented - even with a SELECT
query if a function is executed - see the documentation. It might make sense to highlight that a transaction should be used which can be aborted if the user wants to avoid this.
orf
Oct 30, 2017
Author
Contributor
I didn't consider this, it absolutely should be documented and not the default when verbose=true
!
I didn't consider this, it absolutely should be documented and not the default when verbose=true
!
pope1ni
Oct 31, 2017
Member
Something like this comes to mind:
with transaction.atomic():
transaction.set_rollback(True)
print(queryset.explain(analyze=True))
Something like this comes to mind:
with transaction.atomic():
transaction.set_rollback(True)
print(queryset.explain(analyze=True))
self.query.explain_format, | ||
self.query.explain_verbose) | ||
result.append(prefix) | ||
result.append('SELECT') |
pope1ni
Oct 30, 2017
Member
Does this mean that support for this is limited to SELECT
statements?
- PostgreSQL supports
SELECT
, INSERT
, UPDATE
, DELETE
, and others that are irrelevant to Django...
- MySQL support
SELECT
, INSERT
, UPDATE
, DELETE
, REPLACE
.
Does this mean that support for this is limited to SELECT
statements?
- PostgreSQL supports
SELECT
,INSERT
,UPDATE
,DELETE
, and others that are irrelevant to Django... - MySQL support
SELECT
,INSERT
,UPDATE
,DELETE
,REPLACE
.
orf
Oct 30, 2017
Author
Contributor
I wanted to enable this just for SELECT
queries to begin with. To be honest I'm not sure how I could enable it for the others, .explain()
would have to be added before an .update()/delete()
call and would have to modify the returned values. If you have any ideas how this could be done I would love to hear them, but perhaps we can add this as a separate change.
I wanted to enable this just for SELECT
queries to begin with. To be honest I'm not sure how I could enable it for the others, .explain()
would have to be added before an .update()/delete()
call and would have to modify the returned values. If you have any ideas how this could be done I would love to hear them, but perhaps we can add this as a separate change.
pope1ni
Oct 31, 2017
Member
Agreed. A follow-up ticket to investigate the possibility of this would be a good idea, along with a discussion on the mailing list.
One way is to implement it as you have said, another could be some sort of context manager approach, e.g.
with queryset.explain() as output:
updated = queryset.update(field=value)
print(output)
In this case, updated
ought to be 0
. If queryset.explain(analyze=True)
were used, updated
would be the number of records modified (assuming the backend can report this at the same time).
Agreed. A follow-up ticket to investigate the possibility of this would be a good idea, along with a discussion on the mailing list.
One way is to implement it as you have said, another could be some sort of context manager approach, e.g.
with queryset.explain() as output:
updated = queryset.update(field=value)
print(output)
In this case, updated
ought to be 0
. If queryset.explain(analyze=True)
were used, updated
would be the number of records modified (assuming the backend can report this at the same time).
atombrella
Nov 1, 2017
Contributor
Adding hooks in as_sql
in the different compilers to check whether the query should be executed or explained might be an option? I think it's necessary to do changes in those methods.
Adding hooks in as_sql
in the different compilers to check whether the query should be executed or explained might be an option? I think it's necessary to do changes in those methods.
if self.query.explain_query: | ||
prefix = self.connection.ops.explain_query_prefix( | ||
self.query.explain_format, | ||
self.query.explain_verbose) |
pope1ni
Oct 30, 2017
Member
Hanging indent please, and it should be possible to remove the temporary prefix
.
Hanging indent please, and it should be possible to remove the temporary prefix
.
orf
Nov 11, 2017
Author
Contributor
Done 👍
Done
@@ -154,6 +154,8 @@ Models | |||
~~~~~~ | |||
|
|||
* Models can now use ``__init_subclass__()`` from :pep:`487`. | |||
* The new :meth:`.QuerySet.explain` method allows displaying the execution |
atombrella
Nov 1, 2017
Contributor
Please add a blank line between the *
.
Please add a blank line between the *
.
orf
Nov 11, 2017
Author
Contributor
Done 👍
Done
1b41e90
to
c2f69b9
The ``format`` parameter can output the explanation in different formats, | ||
depending on the database in use. If this parameter is not passed then the | ||
default database format is used, typically text-based. Currently PostgreSQL | ||
supports TEXT, JSON, YAML and XML, while MySQL supports TEXT and JSON. |
adamchainz
Apr 1, 2018
Member
'and' instead of 'while'? Using 'while' implies MySQL is inferior 😱
'and' instead of 'while'? Using 'while' implies MySQL is inferior
orf
Apr 2, 2018
Author
Contributor
:D good point, we would not want people to think that.
:D good point, we would not want people to think that.
if supported_formats: | ||
msg += ' Allowed formats: {0}'.format(', '.join(supported_formats)) | ||
raise ValueError(msg) | ||
|
adamchainz
Apr 1, 2018
Member
What about erroring if options
is non-empty at this point? Subclasses should have already consumed their arguments from it, and if there's anything left it's probably a mistake, like explain(formatt='json')
What about erroring if options
is non-empty at this point? Subclasses should have already consumed their arguments from it, and if there's anything left it's probably a mistake, like explain(formatt='json')
orf
Apr 2, 2018
•
Author
Contributor
I'm a bit on the fence about this, on one hand it would be nice for this to be pretty flexible with what it accepts as I think it's primarily going to be used as a debugging helper in a REPL, but then again if you're passing in options to any backend except Postgres then something is going wrong.
In any case I've made it throw a RuntimeError
, let me know what you think
I'm a bit on the fence about this, on one hand it would be nice for this to be pretty flexible with what it accepts as I think it's primarily going to be used as a debugging helper in a REPL, but then again if you're passing in options to any backend except Postgres then something is going wrong.
In any case I've made it throw a RuntimeError
, let me know what you think
adamchainz
Apr 2, 2018
Member
Best to be conservative and error rather than silently swallow mistyped arguments :) I think ValueError
or TypeError
are more appropriate, they're normally used for argument validation.
Best to be conservative and error rather than silently swallow mistyped arguments :) I think ValueError
or TypeError
are more appropriate, they're normally used for argument validation.
@@ -258,3 +259,21 @@ def window_frame_range_start_end(self, start=None, end=None): | |||
'and FOLLOWING.' | |||
) | |||
return start_, end_ | |||
|
|||
def explain_query_prefix(self, format=None, **options): | |||
prefix = super().explain_query_prefix(format, **options) |
adamchainz
Apr 1, 2018
Member
no need to pass options
up since they are all being consumed here
no need to pass options
up since they are all being consumed here
Thanks for the review, I've made the changes requested. The docs are failing to build but it appears to be an issue in the 'writing-documentation' file of all places, which I've not touched. |
buildbot, test on oracle. |
self.assertTrue(result) | ||
|
||
@skipUnlessDBFeature('supports_explaining_query_execution') | ||
@unittest.skipIf(connection.vendor == 'postgresql', "PostgreSQL specific") |
adamchainz
Apr 2, 2018
Member
Surely 'All backends except PostgreSQL' ? :)
Surely 'All backends except PostgreSQL' ? :)
if supported_formats: | ||
msg += ' Allowed formats: {0}'.format(', '.join(supported_formats)) | ||
raise ValueError(msg) | ||
|
adamchainz
Apr 2, 2018
Member
Best to be conservative and error rather than silently swallow mistyped arguments :) I think ValueError
or TypeError
are more appropriate, they're normally used for argument validation.
Best to be conservative and error rather than silently swallow mistyped arguments :) I think ValueError
or TypeError
are more appropriate, they're normally used for argument validation.
def test_explain(self): | ||
querysets = [ | ||
Tag.objects.filter(name='test').all(), | ||
Tag.objects.filter(name='test').select_related("parent"), |
timgraham
Apr 4, 2018
Member
Use single quotes consistently.
Use single quotes consistently.
Seq Scan on blog (cost=0.00..35.50 rows=10 width=12) | ||
Filter: (title = 'My Blog'::bpchar) | ||
|
||
This method is currently supported by all built in database backends with the |
timgraham
Apr 4, 2018
Member
chop "currently". Is there any plan to add Oracle support? I didn't follow your discussion closely but I thought you basically determined it's infeasible.
chop "currently". Is there any plan to add Oracle support? I didn't follow your discussion closely but I thought you basically determined it's infeasible.
default database format is used, typically text-based. Currently PostgreSQL | ||
supports TEXT, JSON, YAML and XML, and MySQL supports TEXT and JSON. | ||
|
||
Some databases, e.g PostgreSQL, accept flags that can return more |
timgraham
Apr 4, 2018
Member
I think you can chop "e.g PostgreSQL" considering it's mentioned later.
I think you can chop "e.g PostgreSQL" considering it's mentioned later.
|
||
.. warning:: | ||
|
||
This method is only useful to aid debugging poorly performing |
timgraham
Apr 4, 2018
Member
I don't think this should be in a warning box. Put it somewhere in the main text above.
I don't think this should be in a warning box. Put it somewhere in the main text above.
@@ -2476,6 +2476,55 @@ Class method that returns an instance of :class:`~django.db.models.Manager` | |||
with a copy of the ``QuerySet``’s methods. See | |||
:ref:`create-manager-with-queryset-methods` for more details. | |||
|
|||
|
timgraham
Apr 4, 2018
Member
chop blank line
chop blank line
@unittest.skipUnless(connection.vendor == 'postgresql', "PostgreSQL specific") | ||
def test_postgres_explain_options(self): | ||
qs = Tag.objects.filter(name='test').all() | ||
|
timgraham
Apr 4, 2018
Member
remove some blank lines
remove some blank lines
|
||
@unittest.skipUnless(connection.vendor == 'postgresql', "PostgreSQL specific") | ||
def test_postgres_explain_options(self): | ||
qs = Tag.objects.filter(name='test').all() |
timgraham
Apr 4, 2018
Member
is .all()
required?
is .all()
required?
@@ -83,6 +85,99 @@ def setUpTestData(cls): | |||
Cover.objects.create(title="first", item=i4) | |||
Cover.objects.create(title="second", item=cls.i2) | |||
|
|||
@skipIfDBFeature('supports_explaining_query_execution') |
timgraham
Apr 4, 2018
Member
Could you please put these in a new test_explain.py
file? This file is already a bit too long and unorganized.
Could you please put these in a new test_explain.py
file? This file is already a bit too long and unorganized.
|
||
.. warning:: | ||
|
||
On some supported databases these flags can cause the query to be executed |
timgraham
Apr 4, 2018
Member
I'm not sure what "supported databases" means. Is "supported" adding some meaning?
I'm not sure what "supported databases" means. Is "supported" adding some meaning?
@@ -48,6 +48,8 @@ class DatabaseFeatures(BaseDatabaseFeatures): | |||
SET V_I = P_I; | |||
END; | |||
""" | |||
# MySQL has TRADITIONAL instead of TEXT, but we alias TRADITIONAL to TEXT for consistency |
efd2960
to
20ad0c7
Thank you for the review @timgraham, I've made the changes you've requested - they all made sense and I didn't want to reply to each and spam you. After splitting the tests out I found that they failed with the union queryset, it was actually doing entirely the wrong thing (executing the raw query without the explain prefix). I've fixed this and added a test case for There is a missing space in the docs, I'll push a fix for that after the Oracle test suite passes. |
c2c0458
to
475425f
f2b3329
to
5e43070
Silence MySQL warnings in tests? e.g.
|
self.assertIsInstance(result, str) | ||
self.assertTrue(result) | ||
|
||
@unittest.skipIf(connection.vendor == 'postgresql', 'Postgresql gives unrecognized EXPLAIN option "test"') |
timgraham
Apr 17, 2018
Member
3rd party backends might not want this test either. Add a database feature like validates_explain_options
?
3rd party backends might not want this test either. Add a database feature like validates_explain_options
?
I found that something like this will silence the warnings: |
buildbot, test on oracle. |
Thanks @timgraham. I've added the warning filter to |
# Ignore MySQL informational warnings that are triggered with queryset.explain() | ||
warnings.filterwarnings('ignore', '\(1003, *', category=MySQLdb.Warning) | ||
except ImportError: | ||
pass |
pope1ni
Apr 19, 2018
Member
This would be better styled as:
try:
import MySQLdb
except ImportError:
pass
else:
# ...
warnings.filterwarnings(...)
This would be better styled as:
try:
import MySQLdb
except ImportError:
pass
else:
# ...
warnings.filterwarnings(...)
24e3efe
to
460423f
c1c163b
into
django:master
Thanks everyone who helped with this, and thanks for the reviewing @timgraham |
Ticket
I need to add docs, but I hope the general implementation is OK with regards to modifying the
operations
andfeatures
db-backend files, and their interaction with the compiler/queryset/query etc.