fix(Dialog): reset inherited edge signals from ancestor containers

[czarandy]

Summary

• Bug: When XDSDialog is rendered as a DOM descendant of a container that sets edge compensation signals (--edge-start/--edge-end), ghost-variant buttons inside the dialog (like the close button in XDSDialogHeader) incorrectly apply edge compensation negative margins. This happens because CSS custom properties inherit through the DOM tree even for top-layer elements rendered via showModal().
• Root cause: XDSTopNav sets --edge-end: 1 on its endContent wrapper via edgeSignals.end. When a dialog is rendered inside endContent, the close button's edgeCompensation.self style reads the inherited --edge-end: 1 and applies an unwanted -8px margin, making it visually different from dialogs rendered outside the TopNav.
• Fix: Reset --edge-start and --edge-end to 0 on the <dialog> element so edge compensation only applies within intentional container boundaries.

Repro scenario

<XDSTopNav
  endContent={
    <>
      <XDSButton label="Open" onClick={() => setOpen(true)} />
      <XDSDialog isOpen={open} onOpenChange={setOpen} purpose="form" width={900}>
        <XDSLayout
          header={<XDSDialogHeader title="Title" onOpenChange={setOpen} />}
          content={<XDSLayoutContent>Content</XDSLayoutContent>}
        />
      </XDSDialog>
    </>
  }
/>

The close button in the dialog has a visible margin shift compared to an identical dialog rendered outside the TopNav.

Test plan

• Existing Dialog tests pass (22/22)
• Visually verify Dialog close button spacing is identical when rendered inside vs outside TopNav

[czarandy]

Not sure if we have a better way to reset these? LMK

[github-actions]

PR Analysis Report

📚 Storybook Preview

View Storybook for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

🧪 Sandbox Preview

View Sandbox for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

Modified Components

Dialog · View in Storybook

Metric	Before	After	Delta
Bundle Size (ESM)	N/A	752B	N/A
Lines of Code	N/A	397	-
Complexity	N/A	Very High (51)	-

Bundle Size Summary

Package	Size (ESM)	Size (CJS)	Gzipped
@xds/core	14.8KB	23.3KB	3.5KB

Accessibility Audit

Status: No accessibility violations detected.

---

_{Generated by PR Enrichment workflow | Storybook | Sandbox | View full report}

[cixzhang]

Hmm I was thinking to get rid of this kind of edge compensation and then just have the containers that want it adjust the negative margins around the content. But I guess it's tricky for the top nav. That might be the last one I need to think about converting.

[cixzhang]

In the meantime, this is a reasonable fix to unblock.