Reference the source reasoning for each scenario

[notatallshaw]

At the moment it's difficult to validate whether a specific scenario is following the spec or not because it's not referencing it. My idea would be that either in the existing field "explanation", or a new list field that would look something like "sources": [{"url": "...", "direct": true, "notes": "..."}, ...] the source(s) of the scenario can be referenced.

My idea is that:

1. Sources should primarily come from the spec, but in some cases the reasoning may be different, e.g. a comment clarifying the spec on a discussion thread, or copying pip's behavior, etc.
2. "direct": true, would mean the source(s) directly informs the scenario, false would mean this scenario is implied by the source(s) but isn't directly stated, this would apply to almost all cases which involve conflict resolution, as the spec does not cover this directly

This would probably be a lot of work to go back and update every scenario, but if it could be required for new scenarios that it could be slowly fixed over time.

Just a thought anyway, happy to slowly work on it if accepted.

[zanieb]

I'm down to include a reference to the specification in the explanation. Do you need it to be parsable as separate components for some reason or can we just do better about talking about the specification there?

I would be interested in separating the strictly-spec compliant scenarios from the other ones with a directory structure rather than adding more complexity to the schema e.g. scenarios/direct/..., scenarios/indirect/.... I don't really love the direct/indirect names I wonder if we can find something that clearly captures what we mean.

Maybe we should consider directories for each specification e.g. scenarios/pepXXX/...?

[zanieb]

As prompted in #160 maybe we do need something more advanced in the schema like...

expected can become a list (maybe with a compliant flag?) and we can have explanations saying why a different outcome would occur e.g. "uv does blah blah blah".

[notatallshaw]

Do you need it to be parsable as separate components for some reason or can we just do better about talking about the specification there?

Nope, just spitballing ideas.

It's more when reading through this I want to be able to double check why a specific scenario exists, there's a lot of spec and a lot of scenarios, and I can't always keep it in my head.

[zanieb]

Makes sense. I'm hesitant to add it to the schema just to keep things simple for now, but I would definitely appreciate adding references in the expected explanation and/or description as appropriate.

[BurntSushi]

I think we might want even more categories as well. We're also working on multi-platform locking at Astral, and that is going to probably want its own set of distinct scenarios.

I don't yet have a ton of insight here, but my inclination would also be to add some more structure to the scenarios directory. I don't think I have a use case yet for making any schema changes.

[notatallshaw]

I don't yet have a ton of insight here, but my inclination would also be to add some more structure to the scenarios directory. I don't think I have a use case yet for making any schema changes.

FYI, that is discussed in #173, I'm certainly all for it!

[zanieb]

Yeah I think we should consider scenarios/spec/pepXXX.json for the core scenarios and have separate top-level files (or directories if complex) for everything else.