Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update BIOS configuration #129

Closed
7 tasks done
lixuna opened this issue Nov 12, 2018 · 14 comments
Closed
7 tasks done

Update BIOS configuration #129

lixuna opened this issue Nov 12, 2018 · 14 comments
Assignees
@mackonstan @michaelspedersen

Comments

@lixuna
Copy link
Collaborator

lixuna commented Nov 12, 2018 •
edited by pmikus
Loading

  • Add info BIOS settings from CSIT testbed T22
  • Gain access to m2.xlarge.x86 Packet BIOS
  • Add info from Packet system
  • Add suggested changes for BIOS
    • review suggestions. add additional, if needed
  • Determine which systems to update (Eg. all or only packet generators)
  • Update systems with all changes
@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018

CSIT T22 Supermicro BIOS settings:

  • Boot Feature
  |  Quiet Boot                                [Enabled]               |Boot option                  |
  |                                                                    |                             |
  |  Option ROM Messages                       [Force BIOS]            |                             |
  |  Bootup NumLock State                      [On]                    |                             |
  |  Wait For "F1" If Error                    [Enabled]               |                             |
  |  INT19 Trap Response                       [Immediate]             |                             |
  |  Re-try Boot                               [Disabled]              |                             |
  |  Install Windows 7 USB support             [Disabled]              |                             |
  |  Port 61h Bit-4 Emulation                  [Disabled]              |                             |
  |                                                                    |                             |
  |  Power Configuration                                               |                             |
  |  Watch Dog Function                        [Disabled]              |                             |
  |  Restore on AC Power Loss                  [Last State]            |                             |
  |  Power Button Function                     [Instant Off]           |                             |
  |  Throttle on Power Fail                    [Disabled]              |                             |
  • CPU Configuration
  |  Processor Configuration                                           |Enables Hyper Threading      |
  |  --------------------------------------------------                |(Software Method to          |
  |  Processor BSP Revision                    50654 - SKX H0          |Enable/Disable Logical       |
  |  Processor Socket                          CPU1      |  CPU2       |Processor threads.           |
  |  Processor ID                              00050654* |  000506...  |                             |
  |  Processor Frequency                       2.500GHz  |  2.500GHz   |                             |
  |  Processor Max Ratio                            19H  |  19H        |                             |
  |  Processor Min Ratio                            0AH  |  0AH        |                             |
  |  Microcode Revision                        02000030                |                             |
  |  L1 Cache RAM                                  64KB  |      64KB   |                             |
  |  L2 Cache RAM                                1024KB  |    1024KB   |                             |
  |  L3 Cache RAM                               39424KB  |   39424KB   |                             |
  |  Processor 0 Version                                               |                             |
  |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
  |  Processor 1 Version                                               |                             |
  |  Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz                      |                             |
  |                                                                    |                             |
  |  Hyper-Threading [ALL]                     [Enable]                |                             |
  |  Core Disable Bitmap(Hex)                  0                       |                             |
  |  Execute Disable Bit                       [Enable]                |                             |
  |  Intel Virtualization Technology           [Enable]                |                             |
  |  PPIN Control                              [Unlock/Enable]         |                             |
  |  Hardware Prefetcher                       [Enable]                |                             |
  |  Adjacent Cache Prefetch                   [Enable]                |                             |
  |  DCU Streamer Prefetcher                   [Enable]                |                             |
  |  DCU IP Prefetcher                         [Enable]                |                             |
  |  LLC Prefetch                              [Disable]               |                             |
  |  Extended APIC                             [Disable]               |                             |
  |  AES-NI                                    [Enable]                |                             |
  |> Advanced Power Management Configuration                           |                             |
  • Advanced Power Management Configuration
  |  Advanced Power Management Configuration                           |Switch CPU Power Management  |
  |  --------------------------------------------------                |profile                      |
  |  Power Technology                          [Custom]                |                             |
  |  Power Performance Tuning                  [BIOS Controls EPB]     |                             |
  |  ENERGY_PERF_BIAS_CFG mode                 [Maximum Performance]   |                             |
  |> CPU P State Control                                               |                             |
  |> Hardware PM State Control                                         |                             |
  |> CPU C State Control                                               |                             |
  |> Package C State Control                                           |                             |
  |> CPU T State Control                                               |                             |
  • CPU P State Control
  |  CPU P State Control                                               |Enable/Disable EIST          |
  |                                                                    |(P-States)                   |
  |  SpeedStep (Pstates)                       [Disable]               |                             |
  |  EIST PSD Function                         [HW_ALL]                |                             |
  • Hardware PM State Control
  |  Hardware PM State Control                                         |Disable: Hardware chooses a  |
  |                                                                    |P-state based on OS Request  |
  |  Hardware P-States                         [Disable]               |(Legacy P-States)            |
  |                                                                    |Native Mode:Hardware         |
  |                                                                    |chooses a P-state based on   |
  |                                                                    |OS guidance                  |
  |                                                                    |Out of Band Mode:Hardware    |
  |                                                                    |autonomously chooses a       |
  |                                                                    |P-state (no OS guidance)     |
  • CPU C State Control
  |  CPU C State Control                                               |Autonomous Core C-State      |
  |                                                                    |Control                      |
  |  Autonomous Core C-State                   [Disable]               |                             |
  |  CPU C6 report                             [Disable]               |                             |
  |  Enhanced Halt State (C1E)                 [Disable]               |                             |
  • Package C State Control
  |  Package C State Control                                           |Package C State limit        |
  |                                                                    |                             |
  |  Package C State                           [C0/C1 state]           |                             |
  • CPU T State Control
  |  CPU T State Control                                               |Enable/Disable Software      |
  |                                                                    |Controlled T-States          |
  |  Software Controlled T-States              [Disable]               |                             |
  • Chipset Configuration
  |  WARNING: Setting wrong values in below sections may cause         |North Bridge Parameters      |
  |           system to malfunction.                                   |                             |
  |> North Bridge                                                      |                             |
  |> South Bridge                                                      |                             |
  • North Bridge
  |> UPI Configuration                                                 |Displays and provides        |
  |> Memory Configuration                                              |option to change the UPI     |
  |> IIO Configuration                                                 |Settings                     |
  • UPI Configuration
  |  UPI Configuration                                                 |Choose Topology Precedence   |
  |  --------------------------------------------------                |to degrade features if       |
  |  Number of CPU                             2                       |system options are in        |
  |  Number of Active UPI Link                 3                       |conflict or choose Feature   |
  |  Current UPI Link Speed                    Fast                    |Precedence to degrade        |
  |  Current UPI Link Frequency                10.4 GT/s               |topology if system options   |
  |  UPI Global MMIO Low Base / Limit          90000000 / FBFFFFFF     |are in conflict.             |
  |  UPI Global MMIO High Base / Limit         0000000000000000 / ...  |                             |
  |  UPI Pci-e Configuration Base / Size       80000000 / 10000000     |                             |
  |  Degrade Precedence                        [Topology Precedence]   |                             |
  |  Link L0p Enable                           [Disable]               |                             |
  |  Link L1 Enable                            [Disable]               |                             |
  |  IO Directory Cache (IODC)                 [Auto]                  |                             |
  |  SNC                                       [Disable]               |                             |
  |  XPT Prefetch                              [Disable]               |                             |
  |  KTI Prefetch                              [Enable]                |                             |
  |  Local/Remote Threshold                    [Auto]                  |                             |
  |  Stale AtoS                                [Disable]               |                             |
  |  LLC dead line alloc                       [Enable]                |                             |
  |  Isoc Mode                                 [Auto]                  |                             |
  • Memory Configuration
  |                                                                    |POR - Enforces Plan Of       |
  |  --------------------------------------------------                |Record restrictions for      |
  |  Integrated Memory Controller (iMC)                                |DDR4 frequency and voltage   |
  |  --------------------------------------------------                |programming. Disable -       |
  |                                                                    |Disables this feature.       |
  |  Enforce POR                               [Disable]               |                             |
  |  Memory Frequency                          [2666]                  |                             |
  |  Data Scrambling for NVMDIMM               [Auto]                  |                             |
  |  Data Scrambling for DDR4                  [Auto]                  |                             |
  |  tCCD_L Relaxation                         [Auto]                  |                             |
  |  Memory tRWSR Relaxation                   [Enable]                |                             |
  |  2X REFRESH                                [Auto]                  |                             |
  |  Page Policy                               [Auto]                  |                             |
  |  IMC Interleaving                          [2-way Interleave]      |                             |
  |> Memory Topology                                                   |                             |
  |> Memory RAS Configuration                                          |                             |
  • IIO Configuration
  |  IIO Configuration                                                 |Expose IIO DFX devices and   |
  |  --------------------------------------------------                |other CPU devices like PMON  |
  |                                                                    |                             |
  |  EV DFX Features                           [Disable]               |                             |
  |> CPU1 Configuration                                                |                             |
  |> CPU2 Configuration                                                |                             |
  |> IOAT Configuration                                                |                             |
  |> Intel. VT for Directed I/O (VT-d)                                 |                             |
  |> Intel. VMD technology                                             |                             |
  |                                                                    |                             |
  |   IIO-PCIE Express Global Options                                  |                             |
  |  ========================================                          |                             |
  |  PCI-E Completion Timeout Disable          [No]                    |                             |
  • CPU1 Configuration
  |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
  |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
  |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
  |> CPU1 SLOT2 PCI-E 3.0 X16                                          |                             |
  |> CPU1 SLOT4 PCI-E 3.0 X16                                          |                             |
  |> CPU1 SLOT9 PCI-E 3.0 X16                                          |                             |
  • CPU2 Configuration
  |  IOU0 (IIO PCIe Br1)                       [Auto]                  |Selects PCIe port            |
  |  IOU1 (IIO PCIe Br2)                       [Auto]                  |Bifurcation for selected     |
  |  IOU2 (IIO PCIe Br3)                       [Auto]                  |slot(s)                      |
  |> CPU2 SLOT6 PCI-E 3.0 X16                                          |                             |
  |> CPU2 SLOT8 PCI-E 3.0 X16                                          |                             |
  |> CPU2 SLOT10 PCI-E 3.0 X16                                         |                             |
  • South Bridge
  |                                                                    |Enables Legacy USB support.  |
  |  USB Module Version                        17                      |AUTO option disables legacy  |
  |                                                                    |support if no USB devices    |
  |  USB Devices:                                                      |are connected. DISABLE       |
  |        1 Keyboard, 1 Mouse, 1 Hub                                  |option will keep USB         |
  |                                                                    |devices available only for   |
  |  Legacy USB Support                        [Enabled]               |EFI applications.            |
  |  XHCI Hand-off                             [Disabled]              |                             |
  |  Port 60/64 Emulation                      [Enabled]               |                             |
  |  PCIe PLL SSC                              [Disable]               |                             |
  |  Real USB Wake Up                          [Enabled]               |                             |
  |  Front USB Wake Up                         [Enabled]               |                             |
  |                                                                    |                             |
  |  Azalia                                    [Auto]                  |                             |
  |    Azalia PME Enable                       [Disabled]              |                             |
  • PCIe/PCI/PnP Configuration
  |  PCI Bus Driver Version                    A5.01.12                |Enables or Disables 64bit    |
  |                                                                    |capable Devices to be        |
  |  PCI Devices Common Settings:                                      |Decoded in Above 4G Address  |
  |  Above 4G Decoding                         [Enabled]               |Space (Only if System        |
  |  SR-IOV Support                            [Enabled]               |Supports 64 bit PCI          |
  |  MMIO High Base                            [56T]                   |Decoding).                   |
  |  MMIO High Granularity Size                [256G]                  |                             |
  |  Maximum Read Request                      [Auto]                  |                             |
  |  MMCFG Base                                [2G]                    |                             |
  |  NVMe Firmware Source                      [Vendor Defined Fi...]  |                             |
  |  VGA Priority                              [Onboard]               |                             |
  |  CPU1 SLOT2 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
  |  CPU1 SLOT4 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
  |  CPU2 SLOT6 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
  |  CPU2 SLOT8 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
  |  CPU1 SLOT9 PCI-E 3.0 X16 OPROM            [Legacy]                |                             |
  |  CPU2 SLOT10 PCI-E 3.0 X16 OPROM           [Legacy]                |                             |
  |  CPU2 SLOT11 PCI-E 3.0 X4(IN X8) OPROM     [Legacy]                |                             |
  |  M.2 CONNECTOR OPROM                       [Legacy]                |                             |
  |  Onboard LAN1 Option ROM                   [Legacy]                |                             |
  |  Onboard LAN2 Option ROM                   [Disabled]              |                             |
  |  Onboard Video Option ROM                  [Legacy]                |                             |
  |> Network Stack Configuration                                       |                             |
  • ACPI Settings
  |  ACPI Settings                                                     |Enable or Disable Non        |
  |                                                                    |uniform Memory Access        |
  |  NUMA                                      [Enabled]               |(NUMA).                      |
  |  WHEA Support                              [Enabled]               |                             |
  |  High Precision Event Timer                [Enabled]               |                             |
  |  ACPI Sleep State                          [S3 (Suspend to RAM)]   |                             |

@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018

CSIT-TB22 has these additional params configured as GRUB cmdline:

nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable tsc=reliable mce=off

@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018

Captured settings for BIOS of quadtestb-09:
packetnet-quadtestb-09-orig.log

Suggesting following changes:
Sub NUMA Cluster <Disabled> to <Enabled> to skip interleaving of cores
System Profile <Performance> to <Custom> to allow override Turbo Boost <Enabled> to <Disable>

@taylor
Copy link
Collaborator

taylor commented Nov 13, 2018

@michaelspedersen, @mackonstan, please review and add suggestions/thumbs up for changes.

We can push these to all of the reserved instances after @pmikus does a test later today.

@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018 •
edited
Loading

Captured settings for BIOS of quadtestb-09:
packetnet-quadtestb-09-orig.log

Suggesting following changes:
System Profile <Performance> to <Custom> to allow override Turbo Boost <Enabled> to <Disable>

Odd/Even numbering cannot be disabled: http://doc.xueqiu.com/14738e1820f33fea1ba083fd.pdf

@pmikus pmikus closed this as completed Nov 13, 2018
@pmikus pmikus reopened this Nov 13, 2018
@taylor taylor mentioned this issue Nov 13, 2018
Closed
4 tasks
@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018

Output of jitter tool from quadtestb-09 / 147.75.39.241

root@pktgen1:~/pma_tools/jitter# sudo chrt 5 taskset -c 2 ./jitter -i 30
Linux Jitter testing program version 1.9
Iterations=30
The pragram will execute a dummy function 80000 times
Display is updated every 20000 displayUpdate intervals
Thread affinity will be set to core_id:1
Timings are in CPU Core cycles
Inst_Min:    Minimum Excution time during the display update interval(default is ~1 second)
Inst_Max:    Maximum Excution time during the display update interval(default is ~1 second)
Inst_jitter: Jitter in the Excution time during rhe display update interval. This is the value of interest
last_Exec:   The Excution time of last iteration just before the display update
Abs_Min:     Absolute Minimum Excution time since the program started or statistics were reset
Abs_Max:     Absolute Maximum Excution time since the program started or statistics were reset
tmp:         Cumulative value calcualted by the dummy function
Interval:    Time interval between the display updates in Core Cycles
Sample No:   Sample number

   Inst_Min   Inst_Max   Inst_jitter last_Exec  Abs_min    Abs_max      tmp       Interval     Sample No
    160022     171618      11596     160024     160022     171618    1728053248 3205007750          1
    160022     187496      27474     160026     160022     187496    1191182336 3205096604          2
    160022     187496      27474     160024     160022     187496     654311424 3205101560          3
    160022     187538      27516     160026     160022     187538     117440512 3205093410          4
    160022     187508      27486     160024     160022     187538    3875536896 3205103780          5
    160022     187538      27516     160026     160022     187538    3338665984 3205059360          6
    160022     187522      27500     160024     160022     187538    2801795072 3204997190          7
    160022     187470      27448     160024     160022     187538    2264924160 3205052900          8
    160022     187480      27458     160024     160022     187538    1728053248 3205136586          9
    160022     187436      27414     160024     160022     187538    1191182336 3205101690         10
    160022     187484      27462     160024     160022     187538     654311424 3205100624         11
    160022     195316      35294     160024     160022     195316     117440512 3205094318         12
    160022     187538      27516     160026     160022     195316    3875536896 3205117574         13
    160022     194592      34570     160024     160022     195316    3338665984 3205139894         14
    160022     187480      27458     160024     160022     195316    2801795072 3205088646         15
    160022     187424      27402     160024     160022     195316    2264924160 3205115204         16
    160022     187450      27428     160024     160022     195316    1728053248 3205060260         17
    160022     187360      27338     160024     160022     195316    1191182336 3205101616         18
    160022     187538      27516     160026     160022     195316     654311424 3205160644         19
    160022     187480      27458     160024     160022     195316     117440512 3205094050         20
    160022     187476      27454     160024     160022     195316    3875536896 3205115106         21
    160022     187472      27450     160024     160022     195316    3338665984 3205104022         22
    160022     187274      27252     160024     160022     195316    2801795072 3205099508         23
    160022     194882      34860     160024     160022     195316    2264924160 3205181012         24
    160022     187450      27428     167298     160022     195316    1728053248 3205140554         25
    160022     187472      27450     160026     160022     195316    1191182336 3205153784         26
    160022     187418      27396     160024     160022     195316     654311424 3205101706         27
    160022     187268      27246     160024     160022     195316     117440512 3205105530         28
    160022     187406      27384     160024     160022     195316    3875536896 3205150284         29
    160022     187484      27462     160024     160022     195316    3338665984 3205208650         30

@pmikus
Copy link
Collaborator

pmikus commented Nov 13, 2018

MLC outputs from: quadtestb-09 / 147.75.39.241

root@pktgen1:~/mlc/Linux# sudo ./mlc --bandwidth_matrix
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --bandwidth_matrix

Using buffer size of 100.000MB/thread for reads and an additional 100.000MB/thread for writes
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
                Numa node
Numa node            0       1
       0        90935.7 34457.4
       1        34387.1 90870.6

root@pktgen1:~/mlc/Linux# sudo ./mlc --peak_injection_bandwidth
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --peak_injection_bandwidth

Using buffer size of 100.000MB/thread for reads and an additional 100.000MB/thread for writes

Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads        :      181223.5
3:1 Reads-Writes :      172511.5
2:1 Reads-Writes :      172197.1
1:1 Reads-Writes :      174143.7
Stream-triad like:      147942.5

root@pktgen1:~/mlc/Linux# sudo ./mlc --max_bandwidth
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --max_bandwidth

Using buffer size of 100.000MB/thread for reads and an additional 100.000MB/thread for writes

Measuring Maximum Memory Bandwidths for the system
Will take several minutes to complete as multiple injection rates will be tried to get the best bandwidth
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads        :      181234.72
3:1 Reads-Writes :      172514.62
2:1 Reads-Writes :      172199.79
1:1 Reads-Writes :      174149.04
Stream-triad like:      156520.52

root@pktgen1:~/mlc/Linux# sudo ./mlc --latency_matrix
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --latency_matrix

Using buffer size of 2000.000MB
Measuring idle latencies (in ns)...
                Numa node
Numa node            0       1
       0          97.8   149.6
       1         148.9    97.4

root@pktgen1:~/mlc/Linux# sudo ./mlc --idle_latency
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --idle_latency

Using buffer size of 2000.000MB
Each iteration took 207.7 core clocks ( 94.6    ns)

root@pktgen1:~/mlc/Linux# sudo ./mlc --loaded_latency
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --loaded_latency

Using buffer size of 100.000MB/thread for reads and an additional 100.000MB/thread for writes

Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject  Latency Bandwidth
Delay   (ns)    MB/sec
==========================
 00000  165.74   181527.9
 00002  165.72   181528.4
 00008  165.76   181522.3
 00015  164.76   181423.2
 00050  161.03   180458.6
 00100  108.67   110777.2
 00200  102.06    70278.9
 00300  100.32    48476.8
 00400   99.69    37009.2
 00500   99.34    29971.8
 00700   98.45    21790.0
 01000   97.24    15553.0
 01300   96.92    12157.7
 01700   96.71     9479.6
 02500   96.40     6681.5
 03500   96.09     4973.7
 05000   95.76     3688.4
 09000   94.86     2354.8
 20000   93.39     1442.0

root@pktgen1:~/mlc/Linux# sudo ./mlc --c2c_latency
Intel(R) Memory Latency Checker - v3.5
Command line parameters: --c2c_latency

Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT  latency        56.0
Local Socket L2->L2 HITM latency        56.1
Remote Socket L2->L2 HITM latency (data address homed in writer socket)
                        Reader Numa Node
Writer Numa Node     0       1
            0        -   119.0
            1    120.1       -
Remote Socket L2->L2 HITM latency (data address homed in reader socket)
                        Reader Numa Node
Writer Numa Node     0       1
            0        -   194.1
            1    194.0       -

@taylor taylor mentioned this issue Nov 13, 2018
Closed
2 tasks
@pmikus
Copy link
Collaborator

pmikus commented Nov 14, 2018 •
edited
Loading

Operating system

root@pktgen1:~# uname -a
Linux pktgen1 4.4.0-134-generic #160-Ubuntu SMP Wed Aug 15 14:58:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

root@pktgen1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

@pmikus
Copy link
Collaborator

pmikus commented Nov 14, 2018

CPU detail

root@pktgen1:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                56
On-line CPU(s) list:   0-55
Thread(s) per core:    2
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz
Stepping:              4
CPU MHz:               2194.944
BogoMIPS:              4391.31
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              19712K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fx
sr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology n
onstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_
1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single
 intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi
2 erms invpcid rtm cqm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_
llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts flush_l1d

@pmikus
Copy link
Collaborator

pmikus commented Nov 14, 2018 •
edited
Loading

DMIdecode :

Handle 0x1100, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x1000
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: 1
        Locator: A1
        Bank Locator: Not Specified
        Type: DDR4
        Type Detail: Synchronous Registered (Buffered)
        Speed: 2666 MHz
        Manufacturer: 00AD00B300AD
        Serial Number: 11AE6708
        Asset Tag: 01174251
        Part Number: HMA84GR7AFR4N-VK
        Rank: 2
        Configured Clock Speed: 2400 MHz
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

@mackonstan
Copy link
Collaborator

mackonstan commented Nov 14, 2018 •
edited
Loading

Just reviewed the situation with @pmikus and identified following reasons behind calibration data discrepancy vs. FD.io CSIT-18.10:

  1. Gold Skylake 5120 19.25 MB Cache 2.2 GHz vs. Platinum 8180 38.5 MB Cache 2.5 GHz.
  2. DRAM sticks DDR4 2666 MHz, but clocked at 2400 MHz vs. DDR4 2666 MHz.
  3. Ubuntu 16.04 kernel 4.4.0 vs. Ubuntu 18.04 kernel 4.15.0.

@pmikus adjusted identified BIOS parameters, and about to adjust grub parameters.

@Snergster
Copy link
Collaborator

Captured settings for BIOS of quadtestb-09:
packetnet-quadtestb-09-orig.log

Suggesting following changes:
Sub NUMA Cluster <Disabled> to <Enabled> to skip interleaving of cores
System Profile <Performance> to <Custom> to allow override Turbo Boost <Enabled> to <Disable>

ok so small problem...when I follow these changes along with the others (Im assuming its the sub numa cluster change) the output looks different enough that I can only imagine the grub will have to change

basically doubles the number of nodes..

2__root_quadtestb-01___etc_default__ssh__and_update_bios_configuration_ issue__129 _cncf_cnfs

@Snergster
Copy link
Collaborator

ok current state of quad1-4
this results in them all looking like the above

1__ejk_molly__ssh_
1__ejk_molly__ssh_
1__ejk_molly__ssh__and_slack_-_cloud_native_computing_foundation_and_applications

@pmikus
Copy link
Collaborator

pmikus commented Nov 16, 2018

Calibration data.
quad1-calib.log
quad2-calib.log
quad3-calib.log
quad4-calib.log

@pmikus pmikus closed this as completed Nov 16, 2018
@cyring cyring mentioned this issue Dec 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mackonstan mackonstan

@michaelspedersen michaelspedersen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@taylor @Snergster @lixuna @pmikus @mackonstan @michaelspedersen