Increasing development speed: Experimental features

[EwoutH]

Currently, the Mesa ABM project faces the problem that maintenance is fully done by volunteers. Those volunteers have very varying availability, so the time they can spend on the project is not always synchronized between them. This limits development speed, because when one developer has a lot of time, it can take a long time to wait on reviews and releases.

Simultaneously, it's important to keep development speed high and get new potential features to users quickly to be able to get feedback from the community. This requires a quick turnaround on PR merging processes and releases.

Currently, we only have one process: A diligent review. This is because every feature is directly assumed to be API stable and

Therefore, I would like to propose a second grade of features: The introduction of experimental features.

An experimental feature is a new or improved feature that doesn't break backward compatibility. Therefore it can be included even in a patch release, as long as it's marked as experimental and primarily aimed at getting feedback. If uses in production, it should be accepted that behavior can break when upgrading Mesa.

Proposed approach

• Most new features will start as experimental.
• An experimental feature has a clear warning message displayed when released.
  • A discussion thread URL is always added in this message to be able to gather community feedback and track updates and development .
• Experimental features don't follow Semantic Versioning. They can be added, changed or removed in any release, even patch releases.
• Experimental features don't need a diligent review for every change. Changes can be made by a single developer, as long as they are clearly communicated though the whole process.
• To get quick community feedback, patch releases that only contain changes to experimental features or other SemVer changes can be tagged by a single developer.
• Experimental features can be stabilized in any release. From that point onwards, they follow SemVer and lose all the privileges of experimental features.

This not only increases development speed, but it will also increase the robustness of new functionality, since more people can be exposed to it before the feature stabilized, thus gaining wider community support.

Let's be wrong more often, so we can be less wrong in the long-term.

[EwoutH]

I think the easiest and most flexible implementation of this is to just add a warning to any experimental feature (and add it to the docstring).

warnings.warn(
    "The method 'experimental_method' is experimental. It may be changed or removed in any and all future releases, including patch releases.\n
    We would love to hear what you think about this new feature. If you have any thoughts, share them with us here: https://github.com/projectmesa/mesa/discussions/1909"
    FutureWarning,
)

[EwoutH]

Pre-releases is another powerful tool we can use. Pre-releases don’t get installed by accident, but can easily by installed with pip install -U --pre mesa.

[EwoutH]

And to conclude, aside from the warnings, we have the experimental namespace available. I see three questions we should ask:

• How isolated is the feature? (completely (a new module), somewhat (a new method or keyword) or not really (modifications))
• Is it ready to test? (no, only for experience developers, for all users)

I would say the following things:

• For a dev release, it should be ready to test
• Completely new modules should start in experimental
• Dev releases are best used for early testing, gathering feedback or breaking changes
• A warning should be added when we we're exposing it to regular users or want feedback

Maybe you get somethings like this.

mesa-dev-release-matrix.xlsx

But things also can be combined.

[wang-boyu]

Shall we have a dedicated page in ReadtheDocs for all experimental features? This is to introduce cool experimental features (such as #1916) to users. It'll also be easier to migrate from experimental to stable later.

[EwoutH]

@projectmesa/maintainers I think we succeeded quite well on this topic. Contributions are up, release cadence is up, automated testing is more robust (with example testing etc.).

However, I'm really pissed #2006 slipped through the cracks. It happens, but it really shouldn't happen too often.

So, after a bit of thinking I think we need the opposite approach to very stable features than experimental features: we should be very conservative with those. When you touch core components like frequently use schedulers (RandomActivation), the main Grids, or the DataCollector for example, we should be a lot more careful and mostly just test a lot more and wider.

Pre-releases could help, especially if we can find a small pool of people willing to test existing models.

Anyway, I would invite everyone to think about what we can do to make these unfortunate events less likely. I don't think one layer is perfect, so I think we need a swiss cheese model approach. So let's see if we can add some layers or improve some current ones!

[rht]

A more mature test suite (more unit tests and more examples/integration tests) would make experiment easier, and could be a good GSoC project. Even the examples themselves should have test, e.g. Sugarscape with trading.

[jackiekazil]

We could also consider creating more patterns around releases, but this could also slow things down.
For example - there is a target release date and prior to that we do a code freeze and have a period of only testing. (Thinking outloud) I think at this point we are too small for this and are better to stay more nimble.

And +1 to @rht's comment on testing.
I think what is more important was thee response time, and I want to thank you for your advocacy in this space.

[quaquel]

A more mature test suite (more unit tests and more examples/integration tests)

Yes, I completely agree. We have been making good progress already. For example, the benchmarking is a form of integration testing, and we have added tests in response to each identified bug. Of course, there is still room ton futher improve on this but we are moving in the right direction.

I think what is more important was the response time, and I want to thank you for your advocacy in this space.

This was exactly my response while discussing this with @EwoutH in person.

So, after a bit of thinking I think we need the opposite approach to very stable features than experimental features: we should be very conservative with those. When you touch core components like frequently use schedulers (RandomActivation), the main Grids, or the DataCollector for example, we should be a lot more careful and mostly just test a lot more and wider.

I am torn on this. Would we have seen the performance improvements in AgentSet if it had been in experimental? Still, with the new DiscreteSpace stuff, we are working more along the line you suggest. Everything is contained within experimental.