Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto insert sync points #9822

Merged
merged 36 commits into from Dec 14, 2023
Merged

Auto insert sync points #9822

merged 36 commits into from Dec 14, 2023

Conversation

hymm
Copy link
Contributor

@hymm hymm commented Sep 16, 2023

Objective

  • Users are often confused when their command effects are not visible in the next system. This PR auto inserts sync points if there are deferred buffers on a system and there are dependents on that system (systems with after relationships).
  • Manual sync points can lead to users adding more than needed and it's hard for the user to have a global understanding of their system graph to know which sync points can be merged. However we can easily calculate which sync points can be merged automatically.

Solution

  1. Add new edge types to allow opting out of new behavior
  2. Insert an sync point for each edge whose initial node has deferred system params.
  3. Reuse nodes if they're at the number of sync points away.
  • add opt outs for specific edges with after_ignore_deferred, before_ignore_deferred and chain_ignore_deferred. The auto_insert_apply_deferred boolean on ScheduleBuildSettings can be set to false to opt out for the whole schedule.

Perf

This has a small negative effect on schedule build times.

group                                           auto-sync                              main-for-auto-sync
-----                                           -----------                            ------------------
build_schedule/1000_schedule                    1.06       2.8±0.15s        ? ?/sec    1.00       2.7±0.06s        ? ?/sec
build_schedule/1000_schedule_noconstraints      1.01     26.2±0.88ms        ? ?/sec    1.00     25.8±0.36ms        ? ?/sec
build_schedule/100_schedule                     1.02     13.1±0.33ms        ? ?/sec    1.00     12.9±0.28ms        ? ?/sec
build_schedule/100_schedule_noconstraints       1.08   505.3±29.30µs        ? ?/sec    1.00   469.4±12.48µs        ? ?/sec
build_schedule/500_schedule                     1.00    485.5±6.29ms        ? ?/sec    1.00    485.5±9.80ms        ? ?/sec
build_schedule/500_schedule_noconstraints       1.00      6.8±0.10ms        ? ?/sec    1.02      6.9±0.16ms        ? ?/sec

Changelog

  • Auto insert sync points and added after_ignore_deferred, before_ignore_deferred, chain_no_deferred and auto_insert_apply_deferred APIs to opt out of this behavior

Migration Guide

  • apply_deferred points are added automatically when there is ordering relationship with a system that has deferred parameters like Commands. If you want to opt out of this you can switch from after, before, and chain to the corresponding ignore_deferred API, after_ignore_deferred, before_ignore_deferred or chain_ignore_deferred for your system/set ordering.
  • You can also set ScheduleBuildSettings::auto_insert_sync_points to false if you want to do it for the whole schedule. Note that in this mode you can still add apply_deferred points manually.
  • For most manual insertions of apply_deferred you should remove them as they cannot be merged with the automatically inserted points and might reduce parallelizability of the system graph.
  • If you were manually deriving SystemParam, you will need to add system_meta.set_has_deferred if you use SystemParam::apply and want sync points auto inserted after use of your SystemParam.

TODO

  • remove any apply_deferred used in the engine
  • decide if we should deprecate manually using apply_deferred. We'll still allow inserting manual sync points for now for whatever edge cases users might have.
  • Update migration guide
  • rerun schedule build benchmarks

@hymm hymm force-pushed the auto-sync-points branch 2 times, most recently from f69df42 to 810a6d2 Compare September 16, 2023 05:30
@nicopap
Copy link
Contributor

nicopap commented Sep 16, 2023

This is breaking, as users might depend on the deferred behavior of commands. This is known to exists, for example: https://discord.com/channels/691052431525675048/1150574788906065931

@nicopap nicopap added C-Enhancement A new feature A-ECS Entities, components, systems, and events C-Breaking-Change A breaking change to Bevy's public API that needs to be noted in a migration guide C-Usability A simple quality-of-life change that makes Bevy easier to use labels Sep 16, 2023
@nicopap
Copy link
Contributor

nicopap commented Sep 16, 2023

Also this is a regression. I'd love to be able to control whether or not a sync point gets added after my system. For example, by adding a .deferred_apply() combinator to systems to disable automatic insertion

@hymm
Copy link
Contributor Author

hymm commented Sep 16, 2023

Also this is a regression. I'd love to be able to control whether or not a sync point gets added after my system. For example, by adding a .deferred_apply() combinator to systems to disable automatic insertion

Something like this shouldn't be hard to add, but what is the use case here? it's pretty hard to control when commands are applied since apply_deferred applies commands for any system that has run.

@nicopap
Copy link
Contributor

nicopap commented Sep 16, 2023

I was thinking of a system that very rarely spawns entities or insert components. I wouldn't want to pay the overhead of a de-parallelization in such a situation.

@nicopap
Copy link
Contributor

nicopap commented Sep 16, 2023

I think a system like .deferred_apply wouldn't work for the "I want to defer processing" usecase. But it should be fine for a "I want to avoid additional sync points that are not relevant most of the time".

My thinking is that we should have a different Deferred type that applies to a different kind of sync points. But that can wait for a different PR. This one looks good as-is.

@hymm hymm force-pushed the auto-sync-points branch 2 times, most recently from b153620 to 30236fa Compare September 16, 2023 23:34
@hymm
Copy link
Contributor Author

hymm commented Sep 17, 2023

removed any manual sync points in the bevy plugins. Most of the schedules don't have any sync points. A few did or now do.

schedule main pr
Render 5 3
PostUpdate 2 2
SpawnScene 0 1

Render had sync points between each major set. These are now removed, which accounts for the difference there. Need to take a closer look at SpawnScene and see if the added sync point makes sense or not.

@JoJoJet
Copy link
Member

JoJoJet commented Sep 22, 2023

I feel quite torn on this. I've always wanted automatic sync points, but I never envisioned them in this form. I always thought of it being more explicit, where you would express a dependency as system.after_sync_point_after(other), or something similar. That said, your solution certainly seems to be the most elegant. If you think of system.after(other) as meaning "I want system to observe the effects of other", it totally makes sense to include deferred mutations as one of those effects. I'll have to keep chewing on this.

@nicopap
Copy link
Contributor

nicopap commented Sep 22, 2023

Can this be added as an alternative ExecutorKind?

I mean the change is super trivial and very beneficial. Maybe we can add it to 0.12 as a non-default ExecutorKind and consider moving it to the default in 0.13?

@jannik4
Copy link
Contributor

jannik4 commented Sep 22, 2023

I currently have a schedule that needs to be deterministic, therefore I make sure all systems that depend on each other have a before-after relationship (with the help of the ambiguity option). But I still want to apply commands only at the end of the schedule to keep it easier to follow what happens.

So in summary, I think this should be fine as long the ScheduleBuildSettings::auto_insert_sync_points option exist and I can set that to false.

@maniwani
Copy link
Contributor

maniwani commented Sep 22, 2023

I feel quite torn on this. I've always wanted automatic sync points, but I never envisioned them in this form. I've always wanted automatic sync points, but I never envisioned them in this form. I always thought of it being more explicit, where you would express a dependency as system.after_sync_point_after(other), or something similar.

Same. I always thought we'd express these constraints as new DependencyKind variants. Like, imagine a new variant here (named AfterCommandsOf or something).

/// Specifies what kind of edge should be added to the dependency graph.
#[derive(Debug, Clone, Copy, Eq, PartialEq, PartialOrd, Ord, Hash)]
pub(crate) enum DependencyKind {
/// A node that should be preceded.
Before,
/// A node that should be succeeded.
After,
}

And then we could solve them with a sort and sweep (SAP).

As an example, say there are 6 systems, labeled A through F, that have the following constraints.

B.after_commands_of(A)
D.after_commands_of(C)
F.after_commands_of(E)
C.after(A)
C.before(B)
E.after(B)
E.before(D)

To resolve all after_commands_of constraints, all we need is a list of nodes from the after_commands_of edges sorted by their "depth" in the dependency graph. If we collect those pairs into a sorted list of (node, depth, side), we get this.

[(A, 0, left), (C, 1, left), (B, 2, right), (E, 3, left), (D, 4, right), (F, 4, right)]

"Sweeping" through this list, we'd learn that B.after_commands_of(A) and D.after_commands_of(C) can both be satisfied by one sync point and that F.after_commands_of(E) can be satisfied with another.

Visually, that process looks like this.

Expand

(edit) There's a big UX difference between opt-in and opt-out (this PR). If possible, I'd like to see both in action before picking one to merge. (edit: I'm now convinced that opt-out is the better UX. I still wonder if this SAP method I've described would be more efficient at solving the constraints than the PR's current impl is.)

@hymm
Copy link
Contributor Author

hymm commented Sep 22, 2023

opt in vs opt out
If you think of system.after(other) as meaning "I want system to observe the effects of other", it totally makes sense to include deferred mutations as one of those effects.

This is how I think of it. After seeing how many issues users run into in discord with not seeing the effects of their commands in the next system, I think this is the right default. Beginner users shouldn't have to think about when their commands are applied. They just want to see the effects.

Controlling sync points per system is just a micro-optimization. The problem with API's like after_no_sync and no_sync_after is that as soon as you have a apply_deferred somewhere else in the schedule, the after system might run before or it might run after that apply_deferred. So there is no guarantee that the after system will not see the effects of the commands. So the apis don't really let you control when syncs happen, the best they can do is reduce the number of sync points.

I don't think it'd be hard to add a after_no_sync edge in this PR. no_sync_after was just easier since it changed the graph stuff less and provides nearly the same functionality.

Can this be added as an alternative ExecutorKind?

No the changes happen before the executor, during schedule construction. We could change the default of auto_insert_sync_points to false, but I'm not sure I see any reason to do that.

@hymm hymm moved this from Open PR to High Priority PRs in ECS Dec 14, 2023
Copy link
Member

@alice-i-cecile alice-i-cecile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging with our newfound SME-ECS powers!

@alice-i-cecile alice-i-cecile added this pull request to the merge queue Dec 14, 2023
Merged via the queue into bevyengine:main with commit 6b84ba9 Dec 14, 2023
22 checks passed
fn before<M>(self, set: impl IntoSystemSet<M>) -> SystemSetConfigs {
self.into_configs().before(set)
}

/// Run after all systems in `set`.
/// Runs before all systems in `set`. If `set` has any systems that produce [`Commands`](crate::system::Commands)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be "runs after"

@hymm hymm moved this from High Priority PRs to Merged PR in ECS Dec 15, 2023
@hymm hymm deleted the auto-sync-points branch December 28, 2023 00:07
github-merge-queue bot pushed a commit that referenced this pull request Jan 8, 2024
# Objective

Re this comment:
#11141 (comment)

Since #9822, Bevy automatically
inserts `apply_deferred` between systems with dependencies where needed,
so manually inserted `apply_deferred` doesn't to anything useful, and in
current state this example does more harm than good.

## Solution

The example can be modified with removal of automatic `apply_deferred`
insertion, but that would immediately upgrade this example from beginner
level, to upper intermediate. Most users don't need to disable automatic
sync point insertion, and remaining few who do probably already know how
it works.

CC @hymm
github-merge-queue bot pushed a commit that referenced this pull request Jan 10, 2024
# Objective

- Example in error B0003 is not failing anymore after #9822

## Solution

- Update the example code so that is always fail
- Also update logs and instructions on how to debug as it's easier now

---------

Co-authored-by: Alice Cecile <alice.i.cecile@gmail.com>
github-merge-queue bot pushed a commit that referenced this pull request Feb 3, 2024
# Objective

- In #9822 I forgot to disable auto sync points on the Extract Schedule.
We want to do this because the Commands on the Extract Schedule should
be applied on the render thread.
tjamaan pushed a commit to tjamaan/bevy that referenced this pull request Feb 6, 2024
# Objective

- In bevyengine#9822 I forgot to disable auto sync points on the Extract Schedule.
We want to do this because the Commands on the Extract Schedule should
be applied on the render thread.
github-merge-queue bot pushed a commit that referenced this pull request Feb 24, 2024
# Objective

Since #9822, `SimpleExecutor` panics when an automatic sync point is
inserted:

```rust
let mut sched = Schedule::default();
sched.set_executor_kind(ExecutorKind::Simple);
sched.add_systems((|_: Commands| (), || ()).chain());
sched.run(&mut World::new());
```
```
System's param_state was not found. Did you forget to initialize this system before running it?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_ecs::schedule::executor::apply_deferred`!
```

## Solution

Don't try to run the `apply_deferred` system.
msvbg pushed a commit to msvbg/bevy that referenced this pull request Feb 26, 2024
# Objective

Since bevyengine#9822, `SimpleExecutor` panics when an automatic sync point is
inserted:

```rust
let mut sched = Schedule::default();
sched.set_executor_kind(ExecutorKind::Simple);
sched.add_systems((|_: Commands| (), || ()).chain());
sched.run(&mut World::new());
```
```
System's param_state was not found. Did you forget to initialize this system before running it?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_ecs::schedule::executor::apply_deferred`!
```

## Solution

Don't try to run the `apply_deferred` system.
msvbg pushed a commit to msvbg/bevy that referenced this pull request Feb 26, 2024
# Objective

Since bevyengine#9822, `SimpleExecutor` panics when an automatic sync point is
inserted:

```rust
let mut sched = Schedule::default();
sched.set_executor_kind(ExecutorKind::Simple);
sched.add_systems((|_: Commands| (), || ()).chain());
sched.run(&mut World::new());
```
```
System's param_state was not found. Did you forget to initialize this system before running it?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_ecs::schedule::executor::apply_deferred`!
```

## Solution

Don't try to run the `apply_deferred` system.
mockersf pushed a commit that referenced this pull request Feb 27, 2024
# Objective

Since #9822, `SimpleExecutor` panics when an automatic sync point is
inserted:

```rust
let mut sched = Schedule::default();
sched.set_executor_kind(ExecutorKind::Simple);
sched.add_systems((|_: Commands| (), || ()).chain());
sched.run(&mut World::new());
```
```
System's param_state was not found. Did you forget to initialize this system before running it?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_ecs::schedule::executor::apply_deferred`!
```

## Solution

Don't try to run the `apply_deferred` system.
@maniwani maniwani mentioned this pull request Apr 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ECS Entities, components, systems, and events C-Breaking-Change A breaking change to Bevy's public API that needs to be noted in a migration guide C-Enhancement A new feature C-Usability A simple quality-of-life change that makes Bevy easier to use S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it X-Controversial There is active debate or serious implications around merging this PR
Projects
ECS
Merged PR
Development

Successfully merging this pull request may close these issues.

None yet

10 participants