Skylake platform, gcc with CMSSW flags, performance improvements & bug fixes

[dan131riley]

This PR

• Does most of the work of adding Skylake ("SKL") as a supported platform, though currently not enabled by default, and using some hacks for paths on the Cornell system
• Modifies several hit mapping routines to use std::vector<int> instead of std::unordered_map<int,int>. This change assumes the MC hit IDs are (relatively) dense so a vector representation won't require enormous allocations. Improves multiple events in flight scaling on KNL by 50%
• Fixes a FV SelectHitIndices bug and drops fake hit types -3 and -4 from the FV hits-on-track. This improves FV build performance in the transition region to approximately match CE and standard, but doesn't resolve the differences in fit performance.
• Fixes DEBUG compilation errors
• Improves compatibility with gcc with CMSSW options. Changes "pragma simd" to "pragma omp simd", and fixes a variety of picky warning that CMSSW turns into errors

Standard benchmarks here:

https://www.classe.cornell.edu/%7Edsr/mic-track/pr134/

Skylake with icc here:

https://www.classe.cornell.edu/%7Edsr/mic-track/pr134-skylake-icc/

No significant validation differences were observed between SNB and SKL.

[srlantz]

@dan131riley Warning, long digression... The terminology around Skylake is super confusing. SKL can also refer to the Skylake workstation or "Skylake client" chips that do not have AVX-512. We are interested in the"Skylake server" chips that do have AVX-512. These are sometimes (not universally) designated SKX. Species of these newer chips, from 2017, are variously known as "Skylake Xeon", "Skylake-SP", "Intel Xeon processor Scalable family", and probably several other arbitrary marketing names.

Then what's the right abbreviation? I'm guessing the SKX designation came to us via Intel's SDE emulator tool, see https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt . This output tells us that SKX is (or was) a name Intel used internally. I also found an Intel slide deck that refers to an "SKX core" in Skylake-SP: https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf .

WikiChips, on the other hand, just calls all Skylakes SKL, and then they go on to designate 9 subtypes based on the core variants (5 client, 4 server). So in the WikiChips nomenclature, our focus has been on SKL-SP; that's one of the 4 server types with AVX-512. The WikiChips convention appears to be used by AnandTech and Dell too (via a quick Google scan).

Where does that leave us? Nowhere, maybe. SKL seems too broad. SKX is nice and short but isn't very widely used. SKL-SP may be a bit too specific, though it has more currency. Note that GCC's current shorthand for such chips is -march=skylake-avx512; that may be clearest of all, but it's long. There really doesn't seem to be a good abbreviation pertaining only to Skylakes that have AVX-512.

[IHateLinus]

FWIW, I think Boyana’s suggestion to just use the INTEL name (4116 or whatever) seems better than to generate some random names…

…

On Mar 8, 2018, at 12:14 PM, srlantz ***@***.***> wrote: @dan131riley <https://github.com/dan131riley> Warning, long digression... The terminology around Skylake is super confusing. SKL can also refer to the Skylake workstation or "Skylake client" chips that do not have AVX-512. We are interested in the"Skylake server" chips that do have AVX-512. These are sometimes (not universally) designated SKX. Species of these newer chips, from 2017, are variously known as "Skylake Xeon", "Skylake-SP", "Intel Xeon processor Scalable family", and probably several other arbitrary marketing names. Then what's the right abbreviation? I'm guessing the SKX designation came to us via Intel's SDE emulator tool, see https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt <https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt> . This output tells us that SKX is (or was) a name Intel used internally. I also found an Intel slide deck that refers to an "SKX core" in Skylake-SP: https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf <https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf> . WikiChips, on the other hand, just calls all Skylakes SKL, and then they go on to designate 9 subtypes based on the core variants (5 client, 4 server). So in the WikiChips nomenclature, our focus has been on SKL-SP; that's one of the 4 server types with AVX-512. The WikiChips convention appears to be used by AnandTech and Dell too (via a quick Google scan). Where does that leave us? Nowhere, maybe. SKL seems too broad. SKX is nice and short but isn't very widely used. SKL-SP may be a bit too specific, though it has more currency. Note that GCC's current shorthand for such chips is -march=skylake-avx512; that may be clearest of all, but it's long. There really doesn't seem to be a good abbreviation pertaining only to Skylakes that have AVX-512. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8UoZN8w_ZfYcx0czqkvpGbdcWRS9ks5tcZEmgaJpZM4SjG0J>.

[srlantz]

It's not a "random name". Intel's full official name would be Xeon Silver 4116, e.g. However, people commonly refer to the Intel processor generations by abbreviating the lengthier code names: SNB for Sandy Bridge, SKL for Skylake, etc. Unfortunately Intel created different families within Skylake, and we only are interested in the Scalable Processor family, thus SKL-SP. Conclusion: a good name is SKL-SP 4116. Maybe the model number is unique enough, but it gives no context. Steve From: IHateLinus [mailto:notifications@github.com] Sent: Monday, May 07, 2018 5:15 PM To: cerati/mictest Cc: Steve Lantz; Comment Subject: Re: [cerati/mictest] Skylake platform, gcc with CMSSW flags, performance improvements & bug fixes (#134) FWIW, I think Boyana’s suggestion to just use the INTEL name (4116 or whatever) seems better than to generate some random names…

On Mar 8, 2018, at 12:14 PM, srlantz ***@***.******@***.***>> wrote: @dan131riley <https://github.com/dan131riley> Warning, long digression... The terminology around Skylake is super confusing. SKL can also refer to the Skylake workstation or "Skylake client" chips that do not have AVX-512. We are interested in the"Skylake server" chips that do have AVX-512. These are sometimes (not universally) designated SKX. Species of these newer chips, from 2017, are variously known as "Skylake Xeon", "Skylake-SP", "Intel Xeon processor Scalable family", and probably several other arbitrary marketing names. Then what's the right abbreviation? I'm guessing the SKX designation came to us via Intel's SDE emulator tool, see https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt <https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt> . This output tells us that SKX is (or was) a name Intel used internally. I also found an Intel slide deck that refers to an "SKX core" in Skylake-SP: https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf <https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf> . WikiChips, on the other hand, just calls all Skylakes SKL, and then they go on to designate 9 subtypes based on the core variants (5 client, 4 server). So in the WikiChips nomenclature, our focus has been on SKL-SP; that's one of the 4 server types with AVX-512. The WikiChips convention appears to be used by AnandTech and Dell too (via a quick Google scan). Where does that leave us? Nowhere, maybe. SKL seems too broad. SKX is nice and short but isn't very widely used. SKL-SP may be a bit too specific, though it has more currency. Note that GCC's current shorthand for such chips is -march=skylake-avx512; that may be clearest of all, but it's long. There really doesn't seem to be a good abbreviation pertaining only to Skylakes that have AVX-512. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8UoZN8w_ZfYcx0czqkvpGbdcWRS9ks5tcZEmgaJpZM4SjG0J>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub<#134 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHooyoMhtmE_OJaWmzpi87cA0yaWEIS1ks5twLlGgaJpZM4SjG0J>.

[IHateLinus]

I thought SKL 4116 is plenty good enough since no one would know what SP stands for unless you explain further. That is what I called the "random name” part… Basically in either case you will need to specify more if you want to.

…

On May 7, 2018, at 3:49 PM, srlantz ***@***.***> wrote: It's not a "random name". Intel's full official name would be Xeon Silver 4116, e.g. However, people commonly refer to the Intel processor generations by abbreviating the lengthier code names: SNB for Sandy Bridge, SKL for Skylake, etc. Unfortunately Intel created different families within Skylake, and we only are interested in the Scalable Processor family, thus SKL-SP. Conclusion: a good name is SKL-SP 4116. Maybe the model number is unique enough, but it gives no context. Steve From: IHateLinus ***@***.*** Sent: Monday, May 07, 2018 5:15 PM To: cerati/mictest Cc: Steve Lantz; Comment Subject: Re: [cerati/mictest] Skylake platform, gcc with CMSSW flags, performance improvements & bug fixes (#134) FWIW, I think Boyana’s suggestion to just use the INTEL name (4116 or whatever) seems better than to generate some random names… > On Mar 8, 2018, at 12:14 PM, srlantz ***@***.******@***.***>> wrote: > > @dan131riley <https://github.com/dan131riley> Warning, long digression... The terminology around Skylake is super confusing. SKL can also refer to the Skylake workstation or "Skylake client" chips that do not have AVX-512. We are interested in the"Skylake server" chips that do have AVX-512. These are sometimes (not universally) designated SKX. Species of these newer chips, from 2017, are variously known as "Skylake Xeon", "Skylake-SP", "Intel Xeon processor Scalable family", and probably several other arbitrary marketing names. > > Then what's the right abbreviation? I'm guessing the SKX designation came to us via Intel's SDE emulator tool, see https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt <https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt> . This output tells us that SKX is (or was) a name Intel used internally. I also found an Intel slide deck that refers to an "SKX core" in Skylake-SP: https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf <https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf> . > > WikiChips, on the other hand, just calls all Skylakes SKL, and then they go on to designate 9 subtypes based on the core variants (5 client, 4 server). So in the WikiChips nomenclature, our focus has been on SKL-SP; that's one of the 4 server types with AVX-512. The WikiChips convention appears to be used by AnandTech and Dell too (via a quick Google scan). > > Where does that leave us? Nowhere, maybe. SKL seems too broad. SKX is nice and short but isn't very widely used. SKL-SP may be a bit too specific, though it has more currency. Note that GCC's current shorthand for such chips is -march=skylake-avx512; that may be clearest of all, but it's long. There really doesn't seem to be a good abbreviation pertaining only to Skylakes that have AVX-512. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8UoZN8w_ZfYcx0czqkvpGbdcWRS9ks5tcZEmgaJpZM4SjG0J>. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#134 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHooyoMhtmE_OJaWmzpi87cA0yaWEIS1ks5twLlGgaJpZM4SjG0J>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8fN3L6tAB9Bo2f6o_SZb4FN6Oy6Hks5twM-LgaJpZM4SjG0J>.

[srlantz]

As I said before, the SP part is the clue that this is a Skylake server chip with AVX-512, which isn't true of the 5 varieties of Skylake client chips. And of the 4 server types, SP is the only one that can be put in a multi-processor node... Well anyway, when we give presentations, there will probably be enough other clues to let the audience know. So people can do whatever they're comfortable with, I guess. I intend to keep using SKL-SP. Steve

________________________________ From: IHateLinus <notifications@github.com> Sent: Monday, May 7, 2018 8:25 PM To: cerati/mictest Cc: Steve Lantz; Comment Subject: Re: [cerati/mictest] Skylake platform, gcc with CMSSW flags, performance improvements & bug fixes (#134) I thought SKL 4116 is plenty good enough since no one would know what SP stands for unless you explain further. That is what I called the "random name” part… Basically in either case you will need to specify more if you want to.

On May 7, 2018, at 3:49 PM, srlantz ***@***.***> wrote: It's not a "random name". Intel's full official name would be Xeon Silver 4116, e.g. However, people commonly refer to the Intel processor generations by abbreviating the lengthier code names: SNB for Sandy Bridge, SKL for Skylake, etc. Unfortunately Intel created different families within Skylake, and we only are interested in the Scalable Processor family, thus SKL-SP. Conclusion: a good name is SKL-SP 4116. Maybe the model number is unique enough, but it gives no context. Steve From: IHateLinus ***@***.*** Sent: Monday, May 07, 2018 5:15 PM To: cerati/mictest Cc: Steve Lantz; Comment Subject: Re: [cerati/mictest] Skylake platform, gcc with CMSSW flags, performance improvements & bug fixes (#134) FWIW, I think Boyana’s suggestion to just use the INTEL name (4116 or whatever) seems better than to generate some random names… > On Mar 8, 2018, at 12:14 PM, srlantz ***@***.******@***.***>> wrote: > > @dan131riley <https://github.com/dan131riley> Warning, long digression... The terminology around Skylake is super confusing. SKL can also refer to the Skylake workstation or "Skylake client" chips that do not have AVX-512. We are interested in the"Skylake server" chips that do have AVX-512. These are sometimes (not universally) designated SKX. Species of these newer chips, from 2017, are variously known as "Skylake Xeon", "Skylake-SP", "Intel Xeon processor Scalable family", and probably several other arbitrary marketing names. > > Then what's the right abbreviation? I'm guessing the SKX designation came to us via Intel's SDE emulator tool, see https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt <https://software.intel.com/sites/default/files/managed/bf/99/outpt.txt> . This output tells us that SKX is (or was) a name Intel used internally. I also found an Intel slide deck that refers to an "SKX core" in Skylake-SP: https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf <https://www.primeline-solutions.de/files/intel-xeon-scalable-architecture-deep-dive_1.pdf> . > > WikiChips, on the other hand, just calls all Skylakes SKL, and then they go on to designate 9 subtypes based on the core variants (5 client, 4 server). So in the WikiChips nomenclature, our focus has been on SKL-SP; that's one of the 4 server types with AVX-512. The WikiChips convention appears to be used by AnandTech and Dell too (via a quick Google scan). > > Where does that leave us? Nowhere, maybe. SKL seems too broad. SKX is nice and short but isn't very widely used. SKL-SP may be a bit too specific, though it has more currency. Note that GCC's current shorthand for such chips is -march=skylake-avx512; that may be clearest of all, but it's long. There really doesn't seem to be a good abbreviation pertaining only to Skylakes that have AVX-512. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8UoZN8w_ZfYcx0czqkvpGbdcWRS9ks5tcZEmgaJpZM4SjG0J>. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#134 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHooyoMhtmE_OJaWmzpi87cA0yaWEIS1ks5twLlGgaJpZM4SjG0J>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdT8fN3L6tAB9Bo2f6o_SZb4FN6Oy6Hks5twM-LgaJpZM4SjG0J>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub<#134 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHooyt1lxEKpriC14LHF8lquZ6YpEaRHks5twOYTgaJpZM4SjG0J>.