Minimally upgrade blosc 1.7.0 #285
Minimally upgrade blosc 1.7.0 #285
Conversation
|
Er, help! How do I get "/arch:sse2" added to the list of compiler options when compiling under windows? Not via CFLAGS I guess, this from the latest appveyor build...
|
Hmm, perhaps the problem is the output of |
One way to find out... |
...so it looks like |
Hmm tricky. Perhaps this may help: https://docs.python.org/2/library/platform.html |
Here it is a trusty way to detect if the Python interpreter is 32 or 64bit:
|
The appveyor builds now detect 32bit correctly, but the compiler option '/arch:sse2' is being ignored, generating the "SSE2 is not supported by the target architecture/platform and/or this compiler" error. From previous experiments the machine the build is running on is AMD64, I don't know if this makes a difference? Any suggestions? |
This project looks like it's doing a 32-bit build with /arch:sse2, not sure what's different. |
Yes, that's really bizarre (specially because the project you referred to has almost the same setup!). Well, I suppose this is a weirdness of the appveyor environment, so do not spend much time on this. Just tell me when you are done with tests and I will merge. |
The test suite passes reliably, however when running the scripts in the bench/ directory I am getting intermittent segfaults. E.g., running bench/iter.py repeatedly, I will get ~10 runs fine, then a segfault. It is not reliably reproducible, it's intermittent. When recompiling with -O0 or -O1 this problem goes away. I'm sorry I don't have the skills to debug a segfault, so I'll leave this here for now. Just in case we lose the thread, current status of this PR is two unresolved issues:
|
I have spent quite a long time exercising tests and benchmarks on two Linux boxes (Ubuntu 15.04 + gcc 4.9.2 and Gentoo 2.2 + gcc 4.9.3) and I have not being able to see any single glitch. Which platform and compiler are you using? All in all, this looks pretty good to me, so I want to merge soon (unless you think it is a bit green yet). We can catch with the Windows issues later on. |
I'm on Ubuntu 15.10, gcc (Ubuntu 5.2.1-22ubuntu2) 5.2.1 20151010. FWIW I'm in favour of merging, there's nothing else I can add. Would be useful to have in master so others can also run tests and benchmarks. |
Minimally upgrade blosc 1.7.0
This has been merged, but we should dig a bit on what's going on with gcc 5.2.1. @alimanfoo could you please try with a gcc version less than 5.0? Using a clang compiler could be also useful. |
Btw found this in numpy https://github.com/numpy/numpy/blob/master/numpy/distutils/cpuinfo.py has methods for SSE2 detection (although not AVX2 sadly). |
Yes. I wonder how difficult would be adding AVX2 detection to this. 2016-01-14 11:10 GMT+01:00 Alistair Miles notifications@github.com:
Francesc Alted |
I also found https://github.com/workhorsy/py-cpuinfo which may enable AVX2 On Thursday, January 14, 2016, FrancescAlted notifications@github.com
Alistair Miles |
Looks pretty good. A shame that AVX2 and NEON (for ARM) are not there 2016-01-18 11:07 GMT+01:00 Alistair Miles notifications@github.com:
Francesc Alted |
For the record, another possible approach is to try and compile a short spike program, used e.g. here to detect openmp support in the compiler. I don't think this is the right way to go for SSE2 or AVX2 however, as I think you want to know if the CPU on the target system supports the required feature, not just the compiler. Out of my depth here tho, comments welcome. |
After having actually tried it out (and not just reading the code, which is beautiful), it turns out that https://github.com/workhorsy/py-cpuinfo works like a charm and detects sse2 and avx2 (probably neon would be detected on ARM processors too). Provided that py-cpuinfo comes with a MIT license, I think the best solution would be including the |
Awesome! I did ask @workhorsy about AXV2 detection - workhorsy/py-cpuinfo#29 - apparently AVX2 detection via CPUID is not implemented, however CPUID is only one of several methods available for detecting, so if it works via other methods then we're good. |
Sorry for slow response regarding segfaults and gcc. Building current master with gcc-4.9 I see no segfaults. However with gcc 5.2 I still see random intermittent segfaults on my laptop, about 6 in 100 runs of bench/iter.py.
|
Do you have a minimal example that segfaults so that I can have a look? |
Running any of these bench scripts triggers an intermittent segfault: I haven't seen a segfault running the other bench scripts but they are For something minimal, this will trigger a segfault:
...and so will this:
On Wednesday, January 27, 2016, FrancescAlted notifications@github.com
Alistair Miles |
In case it helps..
|
Hmm, can you try with c-blosc in master? I have made some changes since 2016-01-27 11:57 GMT+01:00 Alistair Miles notifications@github.com:
Francesc Alted |
Just fixed this in c-blosc master. Also, I did some cleanup in the chunk extension for bcolz. I cannot get any segfault, but I am using just gcc 4.9.2 and clang 3.6 here. |
OK, thanks, will try on my machine. On Thu, Jan 28, 2016 at 4:26 PM, FrancescAlted notifications@github.com
Alistair Miles |
Still getting segfaults with latest bcolz and c-blosc (38a3e2a and Blosc/c-blosc@c7b6720). |
Uh. Ok will try to setup an Ubuntu box with gcc 5.x and will have a try myself. |
Finally I have setup a Debian box (unstable) and compiled latest c-blosc and in bcolz in master using gcc-5 (Debian 5.3.1-7) 5.3.1 20160121, and I cannot get a single segfault either re-running the benchs lots of times, or the scripts that you attached. For example, for the second example that you pasted:
I am using a vagrant box (via VirtualBox), but SSE2 is enabled there:
So I would say that your segfaults should be related with you compiler version or something else. |
FWIW I've tried this on two different machines with slightly different On Mon, Feb 1, 2016 at 1:07 PM, FrancescAlted notifications@github.com
Alistair Miles |
Yes, the fact that -O0 or -O1 makes the segfaults go away in gcc 5.2 is a good indication that something might be broken with the compiler optimizer. I have even tried with clang 3.7 and 3.8 on my Debian box and I was unable to reproduce the segfault. |
This PR contains the bare minimum required to upgrade c-blosc to 1.7.0.
Test suite passes with 5 tests skipped on my Linux laptop so I don't believe any code changes are required within bcolz itself to accommodate this upgrade.