Fast 1TB swap file for generation

[mstembera]

@syzygy1
Hi Ronald.
Apologies for opening an issue but not sure where best to contact you...

I was wondering if TB generation was CPU speed or memory bandwidth limited? There are now 480GB Intel Optane 900P drives available that could be put into a RAID configuration for a total of almost 1TB and used entirely as a swap file. I wonder if having this 1TB of cheap virtual RAM that may be say 20x slower than normal DRAM would open a genuine possibility of generating 7-men?

Thanks Fisherman

[jjoshua2]

https://www.anandtech.com/show/12163/spotted-15-tb-intel-optane-ssd-900p
So if I got the 960gb optane with my 32 threads and 128gb ram it might work.
I read that ddr3-2133 gets 3 to 4 million iops and optane gets 400k. So generation would take 10x the time roughly. I am curious what queue depth we could get, since 32 threads should get a queue depth of 32 which is enough that RAID should boost performance.

[syzygy1]

@mstembera
Sorry for not responding earlier. Somehow I missed the notification of the opening of this issue.

It seems Optane drives don't wear out nearly as quickly as regular flash drives, and they are faster, so it might be acceptable to use them as swap to get the 1TB virtual RAM necessary for 7-men generation. Generation of a pawnless 7-men TB with the current generator (with a few adaptations to allow 7 men) would require 2 x 462 x 64^5 bytes, which is 924 GiB or 992 GB.

It would make sense though to rewrite the generator before seriously embarking on 7-men generation. Most 7-men pawnless piece combinations have at least two pieces of the same type and color, which can be used to approximately halve the required memory. For 6-piece in-RAM generation it made sense to ignore this (keeping index calculations simpler and faster and total generation time still very acceptable), but for 7-piece generation the situation is different. Also the compression algorithm might have to be revised for greater speed.

[mstembera]

@syzygy1 Thank you for a very informative reply. It looks like Intel might even make DIMMS using 3D Xpoint which I would assume to be very large capacity per $.
https://www.anandtech.com/show/12041/intel-to-launch-3d-xpoint-dimms-in-2h-2018
I will close this since it was simply a question and not an issue.

[jjoshua2]

We could have a lot of 7 piece pawnless tables generated by the end of 2018. No one's board is going to support those dimms. But if someone is building a system early 2019 it could solve them much faster with less latency than nvme. We should have them by 2020 either way. Ddr prices will come down too by 2019.

…

On Dec 16, 2017 6:23 PM, "mstembera" ***@***.***> wrote: @syzygy1 <https://github.com/syzygy1> Thank you for a very informative reply. It looks like Intel might even make DIMMS using 3D Xpoint which I would assume to be very large capacity per $. https://www.anandtech.com/show/12041/intel-to-launch-3d- xpoint-dimms-in-2h-2018 I will close this since it was simply a question and not an issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AO6INE4SidO8V-Uu6AmdUt8iLVMY3oO7ks5tBFD6gaJpZM4Q0BV1> .

[noobpwnftw]

My attempt to generate has started, if it went south may I use this thread to communicate?

I also plan to host direct downloads for all generated files.