Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed #19463 -- Add UUIDField #2923

Closed
wants to merge 1 commit into from
Closed

Fixed #19463 -- Add UUIDField #2923

wants to merge 1 commit into from

Conversation

@mjtamlyn
Copy link
Member

mjtamlyn commented Jul 15, 2014

Added UUIDField. Not particularly complicated this one - no custom lookups or anything. Casts all values to uuid.UUID instances in python.

TODO:

  • More form field tests
  • Documentation

Squashed patch should be marked as fixing #19463.

@timgraham
timgraham reviewed Jul 15, 2014
View changes
django/contrib/postgres/forms/uuid_field.py Outdated
import uuid

from django.core.exceptions import ValidationError
from django import forms

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

I'd alphabetize this before 'django.core'

@timgraham
timgraham reviewed Jul 15, 2014
View changes
docs/ref/contrib/postgres/fields.txt Outdated
.. class:: UUIDField(**options)

A field for storing universally unique identifiiers. Uses PostgreSQL's
built in ``uuid`` data type, and python's :class:`UUID <python:uuid.UUID>`

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

Python's
shorter: ":class:~python:uuid.UUID"

@timgraham
timgraham reviewed Jul 15, 2014
View changes
tests/postgres_tests/test_uuid.py Outdated
class TestSaveLoad(TestCase):

def test_uuid_instance(self):
instance = UUIDModel(field=uuid.uuid4())

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

.objects.create()?

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 15, 2014 Author Member

Not much difference either way I guess. The ArrayField tests were committed with it this way. .create() doesn't really do any magic so I can change them all if you like.

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

I think create() is cleaner, but I wouldn't change the others as part of this PR.

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 15, 2014 Author Member

I'll leave it be for now and then add an extra commit later changing all the tests.

@timgraham
timgraham reviewed Jul 15, 2014
View changes
tests/postgres_tests/test_uuid.py Outdated

def test_invalid_uuid(self):
field = UUIDField()
with self.assertRaises(exceptions.ValidationError) as cm:

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

what does cm stand for?

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 15, 2014 Author Member

context manager?

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

oh, I guess I'm more used to seeing as e for "exception". I see both are used in our tests; thanks.

@timgraham
timgraham reviewed Jul 15, 2014
View changes
tests/postgres_tests/test_uuid.py Outdated

def test_uuid_instance_ok(self):
field = UUIDField()
field.clean(uuid.uuid4(), None) # no error

This comment has been minimized.

@timgraham

timgraham Jul 15, 2014 Member

doesn't clean() return a valid you could check?

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 15, 2014 Author Member

It returns the value, I'm not sure there's any benefit in checking it.

@timgraham
Copy link
Member

timgraham commented Jul 15, 2014

LGTM.

@akaariai
Copy link
Member

akaariai commented Jul 16, 2014

I don't feel that good about adding UUID field for PostgreSQL only - UUID field isn't at all postgresql specific feature. OTOH I do understand why you want to implement it as a PostgreSQL only feature.

Also, UUID field is almost always used as a primary key, so it should be tested as a primary key. The field should get an automatic value when saving a new object into the database (the value can be generated in Python).

I recall a random discussion some time ago on IRC about problems with custom UUID field and admin - if we add UUID field, it should work automatically in admin, too. Unfortunately I don't recall details about this discussion, it was likely about using UUID as a primary key in admin. Maybe we need some simple tests with UUID as primary key in admin?

@akaariai
Copy link
Member

akaariai commented Jul 16, 2014

The simplest solution I can think of for getting automatically a new primary key UUID value when saving a new object is addition of a new method generate_new_pk_value(self, connection) to Field. Then, if it returns something not None, the return value is used as primary key value for the model when inserting a new model. Addition of the generated UUID to the model can be done in Model._save_table(), somewhere around "if not pk_set: fields = [f for f in fields ...].

@schmitch
Copy link
Contributor

schmitch commented Jul 17, 2014

I wouldn't put that in contrib, too. When you look at the code you would see, that its not "that" hard to add to other databases. We would only changed the field based on the connection. Like postgresql uses the db_type uuid, while sqlite3 uses a char field, while mysql uses uuid, too. wouldn't be too hard since the uuid getting created in python and not in the database.

@schinckel
schinckel reviewed Jul 19, 2014
View changes
django/contrib/postgres/fields/uuid_field.py Outdated
from django.utils.translation import ugettext_lazy as _


class UUIDField(six.with_metaclass(SubfieldBase, Field)):

This comment has been minimized.

@schinckel

schinckel Jul 19, 2014 Contributor

I'd suggest an auto keyword to the __init__ for this field type.

Actually, the more I think about it, perhaps it should be even more aggressive:

def get_default(self):
    default = super(UUIDField, self).get_default()
    if not default and not self.null:
        default = uuid.uuid4()
    return default

That is, unless we are allowing NULL, always create a value.

This comment has been minimized.

@schinckel

schinckel Jul 19, 2014 Contributor

Hmm. It occurs to me that if primary_key=True, and we do this, it would look like the object already exists in the database before it actually is saved.

Although, since postgres cannot create UUIDs, that's going to present a problem anyway.

This comment has been minimized.

@schmitch

schmitch Jul 19, 2014 Contributor

Postges can create uuids, when installing the extension.
Am 19.07.2014 15:20 schrieb "Matthew Schinckel" notifications@github.com:

In django/contrib/postgres/fields/uuid_field.py:

@@ -0,0 +1,43 @@
+import uuid
+
+from django.contrib.postgres import forms
+from django.core.exceptions import ValidationError
+from django.db.models import Field, SubfieldBase
+from django.utils import six
+from django.utils.translation import ugettext_lazy as _
+
+
+class UUIDField(six.with_metaclass(SubfieldBase, Field)):

Hmm. It occurs to me that if primary_key=True, and we do this, it would
look like the object already exists in the database before it actually is
saved.

Although, since postgres cannot create UUIDs, that's going to present a
problem anyway.


Reply to this email directly or view it on GitHub
https://github.com/django/django/pull/2923/files#r15143651.

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 22, 2014 Author Member

There is no justification to be this aggressive in the field definition. Making an auto-generating uuid is as simple as UUIDField(default=uuid.uuid4). This should probably be a documented pattern.

This comment has been minimized.

@schinckel

schinckel Jul 24, 2014 Contributor

Yeah, I was wound up on the fact that my similar field was not setting the value correctly on a default, but I was doing something stupid. The default= is a better pattern (and is extensible to dates, etc).

@mjtamlyn
mjtamlyn reviewed Jul 24, 2014
View changes
django/db/models/fields/__init__.py Outdated
@@ -2199,3 +2202,48 @@ def to_python(self, value):
if isinstance(value, six.text_type):
return six.memoryview(b64decode(force_bytes(value)))
return value


class UUIDField(six.with_metaclass(SubfieldBase, Field)):

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 24, 2014 Author Member

I don't honestly understand why this is the only field in here which needs SubfieldBase. I know I'm using to_python but there are other fields here that do so. Perhaps I need to tweak the contribute_to_class? Or is it not an issue?

This comment has been minimized.

@schinckel

schinckel Jul 24, 2014 Contributor

My understanding of SubFieldBase is that all it does is call to_python whenever any value is assigned to the field.

Personally, I wish it were on DateField and friends: being able to pass ISO8601 formatted strings and have them be date/time objects immediately makes for neater test cases.

code='invalid',
params={'value': value},
)
return value

This comment has been minimized.

@schinckel

schinckel Jul 24, 2014 Contributor

Related to comment above: if you passed a non-string value that is not a valid uuid, this would not change the value (or raise an exception), but an invalid string would raise an exception.

I think that's a little inconsistent: but I'm not sure if it's worth worrying about.

This comment has been minimized.

@timgraham

timgraham Sep 4, 2014 Member

Would something like if value and not isinstance(value, uuid.UUID) be better?

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Jul 24, 2014

I've added some tests and notes in the docs about UUIDField(primary_key=True, default=uuid4). I think this is a good enough pattern (and also allows the user to choose a different uuid function if they wish, or none if their application generates uuids client side always. I don't think any extra machinery other than default should be necessary.

As for the admin, I found a number of closed (fixed) tickets relating to non-integer primary keys. If we find any issues with this in the admin I'll happily resolve them.

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Jul 24, 2014

@akaariai @timgraham further review would be appreciated.

@mjtamlyn mjtamlyn changed the title [contrib.postgres] UUIDField Fixed #19463 -- Add UUIDField Jul 24, 2014
@claudep
claudep reviewed Jul 24, 2014
View changes
django/db/models/fields/__init__.py Outdated
super(UUIDField, self).__init__(**kwargs)

def get_internal_type(self):
return "CharField"

This comment has been minimized.

@claudep

claudep Jul 24, 2014 Member

Marc, did you try to store by default in a binary column? AFAIK PostgreSQL is using a binary field internally.

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 24, 2014 Author Member

There's a few issues with using BinaryField:

  • MySQL-Python for python 3 has no support
  • According to the docs they can't be filtered on, which would be a fairly significant flaw
  • They data format stored would be different to that likely used in string representations (i.e. in URLs, json etc) which would be the hex format. This means a user (or Django) would have to convert that string into a uuid.UUID object and then get the binary representation. I guess we might be doing this anyway.
  • Aside from space in the database, I don't see any particular benefit if doing this.

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 24, 2014 Author Member

TBH, I haven't tried it out, it may prove to be a good idea, but I'd want some significant motivation for it.

@schmitch
Copy link
Contributor

schmitch commented Jul 24, 2014

This looks really good. The only things I scared about are Forms with UUIDs, thats somewhat unnormal to enter a UUID as a user. Also the explicit need for default= is somewhat "strange".

And it's too bad that this won't go into 1.7

@mjtamlyn
mjtamlyn reviewed Jul 29, 2014
View changes
django/db/models/fields/__init__.py Outdated
def get_internal_type(self):
return "CharField"

def db_type(self, connection):

This comment has been minimized.

@mjtamlyn

mjtamlyn Jul 29, 2014 Author Member

If this is in core, should it be using internal types instead?

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/forms/fields.txt Outdated
-------------

.. class:: UUIDField(**kwargs)

This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

.. versionadded:: 1.8

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/forms/fields.txt Outdated
This field will accept any string format accepted as the ``hex`` argument
to the :class:`~python:uuid.UUID` constructor.


This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

one space between sections

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/models/fields.txt Outdated

.. class:: UUIDField(**options)

A field for storing universally unique identifiiers. Uses Python's

This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

identifiers

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/models/fields.txt Outdated
@@ -1008,6 +1008,29 @@ Like all :class:`CharField` subclasses, :class:`URLField` takes the optional
:attr:`~CharField.max_length` argument. If you don't specify
:attr:`~CharField.max_length`, a default of 200 is used.


This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

one space between sections

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/models/fields.txt Outdated
---------

.. class:: UUIDField(**options)

This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

.. versionadded:: 1.8

@timgraham
timgraham reviewed Aug 1, 2014
View changes
docs/ref/models/fields.txt Outdated
UUIDField
---------

.. class:: UUIDField(**options)

This comment has been minimized.

@timgraham

timgraham Aug 1, 2014 Member

([max_length=36, **options])

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Aug 5, 2014

Updated fixing docs issues.

@schmitch
Copy link
Contributor

schmitch commented Aug 6, 2014

the build is failing hard due to git errors, not sure why only some are failing..
I've run the failing builts on my box and it failed (I didn't rebased)
so for me this looks merge ready..
Maybe you should rebase manually..

@kezabelle
kezabelle reviewed Aug 6, 2014
View changes
django/db/models/fields/__init__.py Outdated
description = 'Universally unique identifier'

def __init__(self, **kwargs):
kwargs['max_length'] = 36

This comment has been minimized.

@kezabelle

kezabelle Aug 6, 2014 Contributor

Is there a reason that I'm missing (probably) that the field's length is 36 (and each DB's backend definition is 36, with the exception of postgres, which has the native datatype)?
Removing the dashes (if a string) or calling .hex turns the value into 32 chars, which both uuid.UUID() and postgres' datatype support as a input format, and both use the dash-inclusive output format.

This comment has been minimized.

@schmitch

schmitch Aug 6, 2014 Contributor

oh that's what i've missed. thats true there is something wrong on that line

This comment has been minimized.

@mjtamlyn

mjtamlyn Aug 7, 2014 Author Member

Good point, the 36 is with the hyphens. I'll correct that

field.clean(uuid.uuid4(), None) # no error


class TestAsPrimaryKey(TestCase):

This comment has been minimized.

@kezabelle

kezabelle Aug 6, 2014 Contributor

As using UUIDs as primary keys is under test, and thus they may end up used by FKs/M2Ms, it might be prudent to see if ticket 22382 (UUIDs and prefetching) is:

  • valid [I concede it might not be]
  • not a problem in this case, either because of the implementation itself or because of changes elsewhere in Django.

I appreciate that's an extra burden to consider, and isn't directly related to this Field implementation, but should it be a problem, it's probably best caught early in the Field's lifecycle.

This comment has been minimized.

@mjtamlyn

mjtamlyn Aug 7, 2014 Author Member

This is definitely a valid issue, and does apply to this situation. There is a related issue that fields using SubfieldBase do not work as you might expect in values() calls. @akaariai and I have some ideas as to how to solve this and I am planning on working on it. Combining it with this patch though is likely to be confusing.

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Aug 7, 2014

Same project using UUIDField as a pk/"secondary" key/general field at https://github.com/mjtamlyn/uuid-test

It seems the correct signature for a primary key is:
id = UUIDField(primary_key=True, default=uuid.uuid4, editable=False).

@@ -6,12 +6,14 @@
import datetime
import decimal
import math
import uuid

This comment has been minimized.

@collinanderson

collinanderson Aug 7, 2014 Contributor

would it make sense to lazy-load uuid as most projects won't need it? It could be done with just one more line of code in this file and in django/forms/fields.py

This comment has been minimized.

@mjtamlyn

mjtamlyn Aug 7, 2014 Author Member

You could make the same argument for all kinds of modules throughout django - starting with the import of decimal two lines above.

Python's uuid is a single python file. The overhead to import it at server-start time is so negligible it's not worth doing anything.

This comment has been minimized.

@collinanderson

collinanderson Aug 7, 2014 Contributor

It's true, but decimal and datetime are much more popular.

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Aug 16, 2014

Rebased against master.

Should be considered blocked by #3047 so I can remove SubFieldBase

@mjtamlyn mjtamlyn force-pushed the mjtamlyn:dcp-uuid branch Sep 4, 2014
@timgraham
timgraham reviewed Sep 4, 2014
View changes
django/forms/fields.py Outdated
value = super(UUIDField, self).to_python(value)
if value in self.empty_values:
return None
if isinstance(value, six.string_types):

This comment has been minimized.

@timgraham

timgraham Sep 4, 2014 Member

(same pattern as above, if you change it)

@timgraham
timgraham reviewed Sep 4, 2014
View changes
docs/ref/models/fields.txt Outdated
id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
# other fields

Note that the callable is passed to ``default``, not an instance of ``UUID``.

This comment has been minimized.

@timgraham

timgraham Sep 4, 2014 Member

this tripped me up when I first read it. maybe: "Note that a callable (with the parentheses omitted) is passed to default." would be a bit clearer

@timgraham
Copy link
Member

timgraham commented Sep 4, 2014

Please add 1.8 release notes, otherwise LGTM.

@timgraham
Copy link
Member

timgraham commented Sep 4, 2014

model_fields.test_uuid.TestSaveLoad.test_null_handling fails on Oracle.

@mjtamlyn
Copy link
Member Author

mjtamlyn commented Sep 4, 2014

I thought my last commit should fix that (ed970d4)... I'm not sure if the CI did something weird though because the list of commits on the oracle build is not the same as the list on the main build page...

@timgraham
Copy link
Member

timgraham commented Sep 4, 2014

Hm, I don't have insight. Push some changes and try it again.

@mjtamlyn mjtamlyn force-pushed the mjtamlyn:dcp-uuid branch Sep 8, 2014
@mjtamlyn
Copy link
Member Author

mjtamlyn commented Sep 8, 2014

Squashed into a single commit with release notes. IMO, this is now RFC.

@@ -92,7 +92,8 @@ below for information on how to set up your database correctly.
PostgreSQL notes
================

Django supports PostgreSQL 9.0 and higher.
Django supports PostgreSQL 9.0 and higher. It requires the use of Psycopg2
2.0.9 or higher.

This comment has been minimized.

@timgraham

timgraham Sep 15, 2014 Member

Please add to release notes as well.


.. versionadded:: 1.8

.. class:: UUIDField([**options])

This comment has been minimized.

@timgraham

timgraham Sep 15, 2014 Member

should it be [max_length=32, **options]?

This comment has been minimized.

@mjtamlyn

mjtamlyn Sep 16, 2014 Author Member

No, the max_length argument is ignored (or forced) to be 32.

@timgraham
timgraham reviewed Sep 15, 2014
View changes
docs/releases/1.8.txt Outdated
* Django now has a :class:`~django.db.models.UUIDField` for storing
universally unique identifiers. There is a corresponding :class:`form field
<django.forms.UUIDField>`. It is stored as the native ``uuid`` data type on
PostgreSQL, and as a fixed length character field on other backends.

This comment has been minimized.

@timgraham

timgraham Sep 15, 2014 Member

no comma

@timgraham
Copy link
Member

timgraham commented Sep 15, 2014

Ship it!

Uses native support in postgres, and char(32) on other backends.
@mjtamlyn mjtamlyn force-pushed the mjtamlyn:dcp-uuid branch to 1f2745e Sep 16, 2014
@mjtamlyn
Copy link
Member Author

mjtamlyn commented Sep 16, 2014

Merged in ed78212

@mjtamlyn mjtamlyn closed this Sep 16, 2014
@mjtamlyn mjtamlyn deleted the mjtamlyn:dcp-uuid branch Sep 16, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.