Make in-order queues the default via macro

[masterleinad]

This pull request makes in-order queues the default but retains the option for us to change this later easily.
If we are using in-order queues, we don't need any of the depends_on or submit_barrier calls and for benchmarks up to moderate sizes, I see zeCommandListAppendBarrier dominating without this patch.
Related to #6035.

This pull request is in some way reverting back to in-order queues after we lifted the requirement in #5065 (but also guarding a bunch of other synchronization points necessary for out-of-order queues). Thus, this is for performance, not correctness (which was the issue before).

[dalg24]

Make it more obvious in the description that this is in some sort reverting back to using in-order queues. Please add a reference to the previous work (PR link) that had lifted the requirement.

[masterleinad]

Make it more obvious in the description that this is in some sort reverting back to using in-order queues. Please add a reference to the previous work (PR link) that had lifted the requirement.

I add some text in the pull request description.

[dalg24]

Will ignore HIP timeouts. Everything else passed on Jenkins.

[masterleinad]

I just collapsed all the commits into one.

[jedbrown]

Hi 👋. I see 4.1 (https://github.com/kokkos/kokkos/releases/tag/4.1.00) was released without this PR. This continues to be a substantial performance regression for vector operations in PETSc. Was that intentional and is there a time when in-order queues will go in a release?

[PhilMiller]

This was merged to develop after the branch to stabilize the 4.1 release was prepared. It will definitely be in the 4.2 release (barring problems that lead to reversion), and could potentially be included in a hypothetical 4.1.1 release if the argument for it is strong enough.

[jedbrown]

We'll be asking PETSc users to --download-kokkos --download-kokkos-commit=origin/develop (and for KK) if they care about performance, but we'd rather use a release and then users can have simpler version management (especially if they want to install Kokkos themselves instead of via --download-kokkos).

[crtrott]

We could make this part of a 4.1.1