[Bugfix] Disable shared expert overlap if Marlin MoE is used

[mgoin]

Purpose

For now it seems that Marlin MoE might not be safe with multiple CUDA streams, which is triggered when shared expert overlap is used. This was disabled within CI in #28324 so this PR disables shared expert overlap whenever Marlin is used to avoid user issues as well as fix CI.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a bugfix to disable shared expert overlap when Marlin MoE kernels are used. This is achieved by adding a use_marlin_kernels property to the FusedMoE layer, which checks for a use_marlin attribute in the quantization method. The SharedFusedMoE layer is then updated to use this property to conditionally disable overlapping computation. The relevant Marlin-based MoE quantization methods (AWQMoEMethod, CompressedTensorsWNA16MarlinMoEMethod, GPTQMarlinMoEMethod, and Mxfp4MoEMethod) have been correctly updated to set this use_marlin flag. The changes are well-contained and correctly implemented to address the issue. I have no further comments.

[robertgshaw2-redhat]

Can you also revert the disable of overlap from that CI test? Otherwise LGTM

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[vadiklyutiy]

I second the motion

[vadiklyutiy]

But actually my worry is how do we know that other MoE backends is multi streams safe?
It is hard to be 100% sure but maybe we have at least review them and check.
Otherwise such race condition bugs may be really hard to find and identify.

[vadiklyutiy]

it is good if it fails with illegal memory as in this case: at least we know that there is a bug. But frequently race condition might just corrupt hidden state and we got random incorrect output...

[zhewenl]

But actually my worry is how do we know that other MoE backends is multi streams safe?

I agree with this - looks like we are disable this as it's failing in CI, but we need to do a more comprehensive testing for other kernels as well? (especially on older HWs with lesser memory)

cc @mgoin / @vadiklyutiy

[vadiklyutiy]

But actually my worry is how do we know that other MoE backends is multi streams safe?

I agree with this - looks like we are disable this as it's failing in CI, but we need to do a more comprehensive testing for other kernels as well? (especially on older HWs with lesser memory)

cc @mgoin / @vadiklyutiy

My experience says that just comprehensive testing isn't enough (bugs are rare and random). The multi streams part of code should be design and reviewed to be stream safe.