Skip to content

Fix binding of co-locales to architectural features #27243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Fix binding of co-locales to architectural features #27243

wants to merge 9 commits into from
+416 −165

Conversation

jhh67
Copy link
Contributor

@jhh67 jhh67 commented May 15, 2025 •
edited
Loading

Co-locales should be bound to architectural features (such as sockets) only if those features have the same (non-zero) number of PUs. For example, the Apple M3 Pro has two L2 caches, one with only efficiency cores and one with only performance cores. By default, the Chapel runtime only uses performance cores, so if co-locales are bound to L2 caches then one won't have any PUs. This commit corrects the problem by checking that all features of the same type have the same (non-zero) number of PUs.

Resolves #27220.

@jhh67
Fix binding of co-locales to architectural features
a871c93 
Co-locales should be bound to architectural features (such as sockets) only if
those features have the same (non-zero) number of PUs. For example, the Apple
M3 Pro has two L2 caches, one with only efficiency cores and one with only
performance cores. By default, the Chapel runtime only uses performance
cores, so if co-locales are bound to L2 caches then one won't have any PUs.
This commit corrects the problem by checking that all features of the same
type have the same (non-zero) number of PUs.

Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jhh67 jhh67 requested a review from jabraham17 May 15, 2025 17:21
@jhh67 jhh67 mentioned this pull request May 15, 2025
Open
@jhh67
Fix uninitialized variable warning
b6505e9 
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jabraham17
Copy link
Member

Here is the XML of my machine topography for a test case

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc2.dtd">
<topology version="2.0">
  <object type="Machine" os_index="0" cpuset="0x000003ff" complete_cpuset="0x000003ff" allowed_cpuset="0x000003ff" nodeset="0x00000001" complete_nodeset="0x00000001" allowed_nodeset="0x00000001" gp_index="1">
    <info name="Backend" value="Darwin"/>
    <info name="OSName" value="Darwin"/>
    <info name="OSRelease" value="24.4.0"/>
    <info name="OSVersion" value="Darwin Kernel Version 24.4.0: Fri Apr 11 18:33:47 PDT 2025; root:xnu-11417.101.15~117/RELEASE_ARM64_T6000"/>
    <info name="HostName" value="HPE-D2RLK75V43"/>
    <info name="Architecture" value="arm64"/>
    <info name="hwlocVersion" value="2.12.0"/>
    <info name="ProcessName" value="lstopo"/>
    <object type="Package" os_index="0" cpuset="0x000003ff" complete_cpuset="0x000003ff" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="2">
      <info name="CPUModel" value="Apple M1 Pro"/>
      <object type="NUMANode" os_index="0" cpuset="0x000003ff" complete_cpuset="0x000003ff" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="36" local_memory="3624288256">
        <page_type size="16384" count="0"/>
      </object>
      <object type="L2Cache" cpuset="0x00000003" complete_cpuset="0x00000003" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="17" cache_size="4194304" depth="2" cache_linesize="128" cache_associativity="0" cache_type="0">
        <object type="L1Cache" cpuset="0x00000001" complete_cpuset="0x00000001" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="15" cache_size="65536" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000001" complete_cpuset="0x00000001" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="13" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="0" cpuset="0x00000001" complete_cpuset="0x00000001" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="3">
              <object type="PU" os_index="0" cpuset="0x00000001" complete_cpuset="0x00000001" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="37"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000002" complete_cpuset="0x00000002" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="16" cache_size="65536" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000002" complete_cpuset="0x00000002" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="14" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="1" cpuset="0x00000002" complete_cpuset="0x00000002" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="4">
              <object type="PU" os_index="1" cpuset="0x00000002" complete_cpuset="0x00000002" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="38"/>
            </object>
          </object>
        </object>
      </object>
      <object type="L2Cache" cpuset="0x0000003c" complete_cpuset="0x0000003c" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="34" cache_size="12582912" depth="2" cache_linesize="128" cache_associativity="0" cache_type="0">
        <object type="L1Cache" cpuset="0x00000004" complete_cpuset="0x00000004" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="26" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000004" complete_cpuset="0x00000004" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="18" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="2" cpuset="0x00000004" complete_cpuset="0x00000004" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="5">
              <object type="PU" os_index="2" cpuset="0x00000004" complete_cpuset="0x00000004" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="39"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000008" complete_cpuset="0x00000008" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="27" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000008" complete_cpuset="0x00000008" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="19" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="3" cpuset="0x00000008" complete_cpuset="0x00000008" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="6">
              <object type="PU" os_index="3" cpuset="0x00000008" complete_cpuset="0x00000008" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="40"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000010" complete_cpuset="0x00000010" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="28" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000010" complete_cpuset="0x00000010" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="20" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="4" cpuset="0x00000010" complete_cpuset="0x00000010" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="7">
              <object type="PU" os_index="4" cpuset="0x00000010" complete_cpuset="0x00000010" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="41"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000020" complete_cpuset="0x00000020" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="29" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000020" complete_cpuset="0x00000020" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="21" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="5" cpuset="0x00000020" complete_cpuset="0x00000020" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="8">
              <object type="PU" os_index="5" cpuset="0x00000020" complete_cpuset="0x00000020" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="42"/>
            </object>
          </object>
        </object>
      </object>
      <object type="L2Cache" cpuset="0x000003c0" complete_cpuset="0x000003c0" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="35" cache_size="12582912" depth="2" cache_linesize="128" cache_associativity="0" cache_type="0">
        <object type="L1Cache" cpuset="0x00000040" complete_cpuset="0x00000040" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="30" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000040" complete_cpuset="0x00000040" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="22" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="6" cpuset="0x00000040" complete_cpuset="0x00000040" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="9">
              <object type="PU" os_index="6" cpuset="0x00000040" complete_cpuset="0x00000040" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="43"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000080" complete_cpuset="0x00000080" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="31" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000080" complete_cpuset="0x00000080" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="23" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="7" cpuset="0x00000080" complete_cpuset="0x00000080" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="10">
              <object type="PU" os_index="7" cpuset="0x00000080" complete_cpuset="0x00000080" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="44"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000100" complete_cpuset="0x00000100" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="32" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000100" complete_cpuset="0x00000100" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="24" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="8" cpuset="0x00000100" complete_cpuset="0x00000100" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="11">
              <object type="PU" os_index="8" cpuset="0x00000100" complete_cpuset="0x00000100" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="45"/>
            </object>
          </object>
        </object>
        <object type="L1Cache" cpuset="0x00000200" complete_cpuset="0x00000200" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="33" cache_size="131072" depth="1" cache_linesize="128" cache_associativity="0" cache_type="1">
          <object type="L1iCache" cpuset="0x00000200" complete_cpuset="0x00000200" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="25" cache_size="196608" depth="1" cache_linesize="128" cache_associativity="0" cache_type="2">
            <object type="Core" os_index="9" cpuset="0x00000200" complete_cpuset="0x00000200" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="12">
              <object type="PU" os_index="9" cpuset="0x00000200" complete_cpuset="0x00000200" nodeset="0x00000001" complete_nodeset="0x00000001" gp_index="46"/>
            </object>
          </object>
        </object>
      </object>
    </object>
    <object type="OSDev" gp_index="47" name="opencl0d0" subtype="OpenCL" osdev_type="5">
      <info name="Backend" value="OpenCL"/>
      <info name="OpenCLDeviceType" value="GPU"/>
      <info name="GPUVendor" value="Apple"/>
      <info name="GPUModel" value="Apple M1 Pro"/>
      <info name="OpenCLPlatformIndex" value="0"/>
      <info name="OpenCLPlatformName" value="Apple"/>
      <info name="OpenCLPlatformDeviceIndex" value="0"/>
      <info name="OpenCLComputeUnits" value="16"/>
      <info name="OpenCLGlobalMemorySize" value="11184816"/>
    </object>
  </object>
  <support name="discovery.pu"/>
  <support name="discovery.numa"/>
  <support name="discovery.cpukind_efficiency"/>
  <support name="custom.exported_support"/>
  <cpukind cpuset="0x00000003" forced_efficiency="0">
    <info name="DarwinCompatible" value="apple,icestorm;ARM,v8"/>
  </cpukind>
  <cpukind cpuset="0x000003fc" forced_efficiency="1">
    <info name="DarwinCompatible" value="apple,firestorm;ARM,v8"/>
  </cpukind>
</topology>

jabraham17
jabraham17 approved these changes May 22, 2025
View reviewed changes
Copy link
Member

@jabraham17 jabraham17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code changes look good, this just needs the test case and its good to go

jhh67 added 7 commits May 22, 2025 14:43
@jhh67
Changed binding to architectural features
498c3fd 
Changed the logic in the previous commit so that co-locales are bound to an
architectural feature if the number of non-empty instances of that feature
equals the number of co-locales, and if the non-empty instances have an equal
number of cores. For example, the Apple M1Pro has three L2 caches, one of
which has efficiency cores and other two have performance cores. With this
change "-nl 1x2" will bind the two co-locales to the two L2 caches with
performance cores, and ignore the L2 cache that has efficiency cores.

Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jhh67
WIP
a427ba2
@jhh67
Apple M1Pro topology
dd2d2df 
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jhh67
Don't select for non-existent GPUs
e08ecd9 
Don't ask the topology layer which GPUs the locale should use when there
are no GPUs.

Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jhh67
Improve debug messages
5225ff2 
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
@jhh67
WIP
86a5829
@jhh67
WIP
d11cee6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jabraham17 jabraham17 jabraham17 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Attempting to run 3 colocales using the smp substrate crashes
2 participants
@jhh67 @jabraham17