Please revert the change to ListSerializer from #9837 and offer a deprecation path for the way responses come back

[MakotoTheKnight]

I've been waiting a bit for the release of 3.18 to help allow us upgrade to Django 6.1, but we've encountered a potential breaking bug for how the response comes back. Notably, the change is in #9837.

In short, changing from lists to dicts makes perfect sense from a compatibility and alignment standpoint, but because these are different data structures, if we pass these along to an upstream UI on their own (and that UI then iterates over errors, for instance), then we have to update a lot of upstream APIs.

I recognize that this is still the right way to do it, but we need more time than a minor version to upgrade code. I'm maintaining a lot of repositories that leverage ListSerializer which would be impacted by this, which would induce a lot of extra work on our end.

Could this change be reverted or made at least opt-in until a major release instead?

[MakotoTheKnight]

I had noticed a reply to this discussion, and I was set to respond to it in detail, but I'll reiterate my concern that this is a major breaking change, and should be handled in a more structured way to let us migrate to a better pattern:

• Mark the old path as deprecated
• Allow for a configuration flag to use the new code path to allow us to validate changes against our systems
• Remove the old path in a future major release

Having this change in a minor or a patch version would throw off a lot of people who aren't expecting a breaking change to come up in this fashion. Not every project needs to adhere to SemVer, and I don't see anywhere that states that this one is, but given that our team was anticipating a compatibility fix for Django 6.1 in #9978, we could be seen as understandably confused and a bit frustrated about this.

I'm hoping that we can get somewhere on this soon; a non-trivial amount of work would have to go in to ensure compatibility with the error formatting in this serializer that we didn't anticipate would be coming up.

[browniebroke]

I've suspected that this change might be disruptive when we merged it but went ahead anyway as it felt like an improvement. I considered doing a deprecation but felt like it was a kind of change that would be difficult to support 2 behaviour, but maybe that was lazy on my part...

Having this change in a minor or a patch version would throw off a lot of people who aren't expecting a breaking change to come up in this fashion.

3.18 is NOT a minor change, we're never going to release a version 4 of DRF and when we bump the 2nd digit in the version, there ARE breaking changes, which are clearly outlined in the release notes, if these people would care to read it.

Now, that begs the question whether our versioning scheme is worth revising, (especially as Django is moving to calendar versioning). I think we could communicate expectations better and avoid users confusion. When a project doesn't use SemVer, it's often better to not have version numbers that looks like SemVer, which avoid these kind of conversations entirely...

I'm hoping that we can get somewhere on this soon; a non-trivial amount of work would have to go in to ensure compatibility with the error formatting in this serializer that we didn't anticipate would be coming up.

Adding a deprecation would delay the pain, and I'm a bit concerned that we would hit the same issue when we eventually remove the deprecation.

What kind of compatibility layer would you expect? I get that you want to be able to opt-in the new shape, but how would you want to control this? A project setting? How is it different than updating DRF and fixing it as part of the upgrade?

[MakotoTheKnight]

Thanks for the thorough and thoughtful response to this! I totally get that this project doesn't leverage SemVer, so with that expectation set now I can move past that part.

What kind of compatibility layer would you expect? I get that you want to be able to opt-in the new shape, but how would you want to control this?

I'm planning on meeting with the rest of my department to see how broad the impact is and what we might want to see as a resolution to this, but I can speak to the initial scope of the change right now. As way of splitballing, I see:

• Creating a new ListSerializer that contains the "correct" error reporting shape, which could be monkey-patched in based on a config setting
• Raising DeprecationWarning to the old ListSerializer to give us context and identify places where we need to change/update (using many=True to find occurrences is still a bit of a shotgun approach)

I can speak more concisely to the impact, which is why I reached out:

Today, we have a minimum of 30 microservices which leverage the ListSerializer. All of these services have upstream consumers of this data and its outputs with errors. What we're not 100% clear on is how those upstream consumers use the error output. There would be a lot of work to both:

• Identify where the error is consumed, and
• what the upstream consumers are expecting in terms of a shape; some could be using the response code, some could be rendering the string, some could be doing both or neither. It's a big project.

How is it different than updating DRF and fixing it as part of the upgrade?

I think the main logjam is that we're between Django and DRF versions right now. Django 6.1 and DRF 3.17 both contain CVEs which are rated at a "high" or "medium", and our current development policies require us to act on both of those findings on an expedited timeline. We can't upgrade just one or the other since there is a breaking change within Django itself that makes > 6.1 incompatible with DRF < 3.18.

In other words, to be able to upgrade to address CVEs in Django > 6.1, DRF needed to become compatible with Django > 6.1, but with that compatibility fix we also have a huge refactoring of how errors are propagated.

Let me know if I can clarify anything else. I'll reply again later once I hear back from our group.

[browniebroke]

My question on the compatibility layer was to inform what kind of hooks should DRF think about in this instance. I can see 2 levels of triggering this: either via a project-wide setting or via an option at the serializer level. The second is more granular, but probably more complex to implement.

Creating a new ListSerializer that contains the "correct" error reporting shape, which could be monkey-patched in based on a config setting

I don't think we need a separate serializer, all the changes are in a single method, so one could branch into that method. You could even do it yourself to be fair: fork DRF, make the changes yourself on top of 3.18 and run with that while you migrate your systems. How long do you expect such migration to take? I guess this won't be a high priority effort...

Today, we have a minimum of 30 microservices which leverage the ListSerializer. All of these services have upstream consumers of this data and its outputs with errors. What we're not 100% clear on is how those upstream consumers use the error output.

I can see the problem you're having with the amount of systems to update, and interdependencies between them.

I think the main logjam is that we're between Django and DRF versions right now. Django 6.1 and DRF 3.17 both contain CVEs which are rated at a "high" or "medium", and our current development policies require us to act on both of those findings on an expedited timeline. We can't upgrade just one or the other since there is a breaking change within Django itself that makes > 6.1 incompatible with DRF < 3.18.

I don't follow this point, we released 3.17.2 precisely to fix some vulnerabilities ahead of 3.18, which we knew would be more disruptive. You should have the same level of security fixes in 3.17.2 than there is 3.18.0. Similarly, AFAIK Django 6.0.x is still receiving security patches and I don't think you need to upgrade to 6.1 to get a security fix, they're all in the latest 6.0.x as well.

[MakotoTheKnight]

My question on the compatibility layer was to inform what kind of hooks should DRF think about in this instance. I can see 2 levels of triggering this: either via a project-wide setting or via an option at the serializer level. The second is more granular, but probably more complex to implement.

Sure - I can address that more concisely. I think it'd be better to go at the project level instead.

I don't think we need a separate serializer, all the changes are in a single method, so one could branch into that method. You could even do it yourself to be fair: fork DRF, make the changes yourself on top of 3.18 and run with that while you migrate your systems. How long do you expect such migration to take? I guess this won't be a high priority effort...

I don't think we have the bandwidth to support a fork of the code temporarily, but I won't rule it out yet.

It wouldn't be a high priority effort, but it would be something we could get to as our high priority bucket goes down.

I don't follow this point, we released 3.17.2 precisely to fix some vulnerabilities ahead of 3.18, which we knew would be more disruptive. You should have the same level of security fixes in 3.17.2 than there is 3.18.0.

We've unpinned our dependencies, which - in practice - may not be the best approach, but it's how we're able to identify incompatibilities with newer versions of code. This also allows us to identify and fix vulnerabilities that are resolved with an upgrade fairly quickly.

3.17.2 isn't compatible with Django < 6.1 due to cc_delim_re still being imported; if that were in 3.17.x then we might be able to wait for a solution on 3.18.x.