How can I integrate pydantic v2 URLs in code?

[bunny-therapist]

In pydantic v1, I could set a field to be "AnyHttpUrl", and the format would be validated. The field value would be of type AnyHttpUrl, but that itself is ultimately a str. However, in pydantic v2, instead the field value would be of type "pydantic_core._pydantic_core.Url'" which is not a str. In other words, the following would print "True" in pydantic v1 but "False" in pydantic v2:

import pydantic

class MyModel(pydantic.BaseModel):
    url: pydantic.AnyHttpUrl

x = MyModel(url="http://google.com")

print(isinstance(x, str))

In pydantic v1, the url is a str, which means it has all the methods of a str and passes all isinstance checks of a str. In pydantic v2, it is of a type which is an internal pydantic class. I have therefore no idea how to integrate this in my code. Consider the following example code:

import pydantic
import requests

class MyModel(pydantic.BaseModel):
    url: pydantic.AnyHttpUrl


def get_from_url(url: str) -> requests.Response:
    return requests.get(url)

x = MyModel(url="http://google.com")
response = get_from_url(url=x.url)

In pydantic v1, this would just work. I have function which takes a str, so I pass in x.url to it. However, in pydantic v2, x.url is NOT a str anymore, it is actually of a type which is internal to pydantic. Therefore it does not seem feasible to use it in type hints, and I cannot really use its methods (I mean, it is a private class, I should not have to know its methods - my type checker/IDE certainly does not know). It seems like the only thing I can do is to convert x.url to str - str(x.url) - before using it - every single time I access the field.

How are issues like this supposed to be resolved? Am I supposed to never access fields without converting them?

(I ran into this issue when trying to go from pydantic v1 to v2, and I got lots of errors saying, e.g., AttributeError: 'pydantic_core._pydantic_core.Url' object has no attribute 'startswith' and similar. My code is obviously designed to handle the URLs as strings.)

[g0di]

I was about to start the exact same conversation before bumping on yours. The thing is, I'm wondering if this was even expected because I could not find any mention of this change neither in the doc nor the migration guide nor the future plan.

I'm supporting the fact that having URLs inherit from str was really useful and IMO made totally sense (because an URL is ultimately a string). Moreover, most tools/libraries dealing with URLs expect and use str in place of URLs so this force to have to wrap all our pydantic Url with str(my_url).

This is not actually a big deal to me but I'm really wondering the reason behind (if any) and also to have this mentionned somewhere

[bunny-therapist]

A small correction: since pydantic.AnyHttpUrl has a defined python stub interface, type hinting using it actually works.

However, I am still stuck either calling str(...) or .unicode_string() on it everywhere (not sure which one of the two is best-practice, or if there is a difference between them). It is certainly less practical than when they were strings.

[mzamini92]

pydantic.AnyHttpUrl field has a defined Python stub interface but you still need to convert it to a str before you can use it in most other ways. If you just need to convert the field to a string that you can pass to another function, then the str() function is probably the best choice. If you need to access the underlying Unicode string representation of the URL, then the .unicode_string() method is the best choice. Here is an example of how you can use the str() function to convert an AnyHttpUrl field to a str:

import pydantic

class MyModel(pydantic.BaseModel):
    url: pydantic.AnyHttpUrl

x = MyModel(url="http://google.com")

print(str(x.url)) # This will print "http://google.com"

[phdesign]

Using an Annotated Validator to convert the serialised format to str seems to work for me. I'm new to pydantic though so there may be some downsides to this approach that I'm not seeing.

from typing_extensions import Annotated
from pydantic import AfterValidator, BaseModel, HttpUrl

HttpUrlString = Annotated[HttpUrl, AfterValidator(lambda v: str(v))]


class MyModel(BaseModel):
    url: HttpUrlString


x = MyModel(url="http://google.com")

print(x.url)
print(isinstance(x.url, str))

# Output:
# http://google.com/
# True

[morcef]

Not sure this is the case for all IDEs, but PyCharm still sees url as Url | Url in this case. So warnings will be raised when trying to pass it to a function that expects a string.

[benoit-bnp]

I tried this solution (with FastAPI, for an input parameter of a route) but I get the following warning when testing: Expected `url` but got `str` - serialized value may not be as expected. If anyone has an idea to prevent that, I'd gladly take it.

[awwitecki]

One of solutions could be declaring a field as str and validate it manually though for sure it's not necessarily elegant:

from typing import Annotated

from pydantic import BaseModel, BeforeValidator, HttpUrl, TypeAdapter

http_url_adapter = TypeAdapter(HttpUrl)

Url = Annotated[str, BeforeValidator(lambda value: str(http_url_adapter.validate_python(value)))]


class Model(BaseModel):
    url: Url

EDIT: Adapter moved out from Annotated as suggested

[offlinehacker]

This is great, why should URL data format be anything but string after all.

[samuelcolvin]

You should move the type adapter out of the validator for performance.

[ramsrib]

Ran into this bug (which causes the url to be appended with "/") after fixing converting the url to str.

url = "http://google.com"
assert str(HttpUrl(url)) == url # it fails with Pydantic 2.0 - 2.5.3

[kfreezen]

This is one of the many things that is causing me grief in a move from pydantic v1 to pydantic v2.

Now, I love pydantic v2, and its speed increases, but running into stuff like this makes me wish I hadn't moved my project to it. Making changes like these are fine for my code, but then when I have to require other developers to also make these little changes to their code, it causes them heartache and pain.

@samuelcolvin if you would only give us a way to have the old behavior back, I would be eternally grateful. Or at least grateful for the next 70 years.

If I could import something like the following to enable the behavior, it would ease our migration path. Or even a way to implement the old behavior with an equivalent class that we use in our own code. It doesn't matter.

from pydantic.legacy import AnyHttpUrl

Thank you for reading.

[antont]

Have you tried the

HttpUrlString = Annotated[HttpUrl, AfterValidator(lambda v: str(v))]

thing from above? I'm using it, mostly happily IIRC

I'm not saying that the lib should not provide it, but just wondering if that first aid bandage works for you or not .. as am not sure how well it works for me.

[samuelcolvin]

Or indeed simple

HttpUrlString = Annotated[HttpUrl, AfterValidator(str)]

[antont]

Or indeed simple

HttpUrlString = Annotated[HttpUrl, AfterValidator(str)]

Right, I actually saw the lambda version in our codebase recently and was wondering why the do-nothing lambda was there .. but didn't dare to touch it. Thanks!

[lupodellasleppa]

@kfreezen pydantic.v1 actually exists, but if you were to try and use it for a single schema, then you'd have to use the BaseModel from pydantic.v1 as well for that schema, as well as all other pydantic types from pydantic.v1, as the pydantic types and the BaseModel from pydantic v2 are not compatible. This could cause a lot of headache having to remember to use, for example, model_dump() instead of dict(), or model_validate() instead of from_orm(), only for that particular model everyhwere it is used.

Also, this workaround

HttpUrlString = Annotated[HttpUrl, AfterValidator(str)]

produces the warning described in the comment above

The provided solution to this causes mypy errors instead

No overloads for "init" match the provided arguments

Actually, the only way I found it working for now, without errors, warning or type check nuisances, is also the way I think now it's supposed to be used, that is dumping the model using model_dump(mode="json"), which returns a dict of only JSON-serializable types.

Example:

from pydantic import BaseModel, HttpUrl


class MyModel(BaseModel):
    my_url: HttpUrl


my_model = MyModel(my_url="http://www.github.com")
dump_python = my_model.model_dump()
dump_json = my_model.model_dump(mode="json")  # <<< this

print(my_model)
# my_url=Url('http://www.github.com/')
print(dump_python)
# {'my_url': Url('http://www.github.com/')}
print(dump_python["my_url"])
# http://www.github.com/
print(type(dump_python["my_url"]))
# <class 'pydantic_core._pydantic_core.Url'>
print(dump_json)
# {'my_url': 'http://www.github.com/'}
print(dump_json["my_url"])
# http://www.github.com/
print(type(dump_json["my_url"]))
# <class 'str'>

Disclaimer: this particular example produces a mypy warning for

"Literal['http://www.github.com']" is incompatible with "HttpUrl"

but not creating the instance explicitly (for example receiving the model from a FastAPI route) won't display this.