you may need a cast phase3.hpp

[Hardcore-fs]

/Volumes/null/chia-plotter/include/chia/phase3.hpp:48:24: error: no matching function for call to 'min' const auto count = std::min(L_position - L_offset, L_buffer.size()); ^~~~~~~~ /usr/local/opt/llvm/bin/../include/c++/v1/algorithm:2594:1: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned long long' vs. 'unsigned long') min(const _Tp& __a, const _Tp& __b)

correction seems to be:
const auto count = std::min<unsigned long>(L_position - L_offset, L_buffer.size());

[madMAx43v3r]

that's funny, I knew it was risky without a cast ;)

[Hardcore-fs]

I think it all depends on your expectations.
I finally got the thing running on osx but it is breaking badly with an exec 11 in the disk stuff
it writes the files out but as soon as it starts a sort it abends.....
It's like clubbing baby seals....

[madMAx43v3r]

can you debug it and send me stack trace or something? don't have a mac to test on.

[Hardcore-fs]

yep.. don't expect to get it working...
it was more of something to do on a rainy sat.

`

• thread cannot build on osx due to pthread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  • frame #0: 0x00007fff70a9cb66 libsystem_kernel.dylib__pthread_kill + 10 frame #1: 0x00007fff70c67080 libsystem_pthread.dylibpthread_kill + 333
    frame you may need a cast phase3.hpp #2: 0x00007fff709f81ae libsystem_c.dylibabort + 127 frame #3: 0x0000000100e29ca3 libc++abi.1.dylibabort_message + 195
    frame Unclear how to set the key #4: 0x0000000100e0966e libc++abi.1.dylibdemangling_terminate_handler() + 238 frame #5: 0x0000000100e28c68 libc++abi.1.dylibstd::__terminate(void ()()) + 8
    frame Plotter is writing to parent directory of specified working directory #6: 0x0000000100e2be6b libc++abi.1.dylib__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) + 27 frame #7: 0x0000000100e2bde5 libc++abi.1.dylib__cxa_throw + 117
    frame Not compiling on Apple M1 system #8: 0x000000010001db3c test_phase_1Thread<unsigned long long>::take(unsigned long long&) + 156 frame #9: 0x000000010001a3d6 test_phase_1ThreadPool<unsigned long long, std::__1::vector<phase1::entry_1, std::__1::allocatorphase1::entry_1 >, unsigned long>::take(unsigned long long&) + 230
    frame CMake Error at CMakeLists.txt:5 (add_subdirectory): #10: 0x000000010000bf99 test_phase_1Processor<unsigned long long>::take_copy(unsigned long long const&) + 41 frame #11: 0x0000000100003f39 test_phase_1void phase1::compute_f1<DiskSort<phase1::entry_1, phase1::get_yphase1::entry_1 > >(unsigned char const, int, DiskSort<phase1::entry_1, phase1::get_yphase1::entry_1 >*) + 457
    frame P1 bug? #12: 0x000000010000323e test_phase_1main + 782 frame #13: 0x00007fff7094c015 libdyld.dylibstart + 1
    `

Thread 19 Crashed:: Disk/read/0 0 libc++abi.1.0.dylib 0x000000010b63cdc3 __cxa_throw + 83 1 test_phase_1 0x000000010a8f417d DiskSort<phase1::entry_1, phase1::get_y<phase1::entry_1> >::read_bucket(unsigned long&, std::__1::vector<std::__1::vector<phase1::entry_1, std::__1::allocator<phase1::entry_1> >, std::__1::allocator<std::__1::vector<phase1::entry_1, std::__1::allocator<phase1::entry_1> > > >&) + 541 2 ??? 0x0007c25100e38e32 0 + 2183978000027186

[madMAx43v3r]

Ah that's not a crash, it was throwing an exception, there should be some output in the terminal?

[Hardcore-fs]

basically nothing.....
when a mac crashes, all it does it dump all the threads it was processing and prints the above
then produces a dump file.
It's writing out GB of data to 127 files, gets them all to about 301/9mb before it crashes...
which it does just after generating 127, test.p1.t2.sort_bucket files

last program line before ending in kernal traps

[madMAx43v3r]

are you running out of disk space by any chance? or hitting a maximum allowed open file limit?

[Hardcore-fs]

Hi,
nope not on a 6TB drive TB spare!!
launchctl limit maxfiles
maxfiles 256 unlimited

so basically it warns at 256, but it's unlimited....

... i found it , but I cannot figure out WHY!!!!
basically on line 198, when it attempts to read the file it returns zero. (fails and aborts)

I thought perhaps the file pointer was at end of file, so i added a :
rewind(bucket.file);
but still reads zero.
The file is ON the disk , has >300Mb of data...
so i'm currently trying to find out why....

The only other thing i can think of is perhaps the file is not "flushed" closed & reopened..
and so maybe considers it still "write"

Why would a file with 300 MB of data , that can be freely copied and reports the size as:
301,989,546 bytes (302 MB on disk)
think it is zero length....
the only other thing i can think of , is it requires a "full path", but if that were true how can it write it to start with.
anyway i'm looking now..

but you can see i even grab the size of the file & save it is "size"
but "readentries" =0, num_entries is 65536

[madMAx43v3r]

The bucket files are closed in phase1.hpp:400, and re-opened for reading in DiskSort.hpp:183. Even if it wasn't closed beforehand it will be closed before opening again for reading in DiskSort<T, Key>::bucket_t::open().

[madMAx43v3r]

I found a bug that might be the cause of this, fix pending.

[madMAx43v3r]

Try latest master

[Hardcore-fs]

LOL.. extreme bug hunting....
it is something to do with the size of the transfer.....
if i threw in a "while" loop and dropped the size down it worked first time for transferring to the buffer...
Just it was only a dummy test.

latest master does not fix issue the error is still being thrown in "DiskSort"
on the FREAD.

[madMAx43v3r]

If size of transfer is an issue, try to lower the N:
static constexpr size_t N = 65536;
in DiskSort.hpp:192

[Hardcore-fs]

i'm beginning to think its related to "T::disk_size" of 9.
maybe mult "T::disk_size" by a power of 2 it and divide the "N-1" value along teh same lines so the "-1" offset does not break anything.

the machine has 96GB ram.....so it's not a buffer allocation problem...
the other thing is even the FREAD fails, the buffer STILL has file content in it, so i'm thinking some sort of kernel implementation stupidity some place.

[madMAx43v3r]

Try setting it to 10 or 12 or 16, in phase1.h:36
It will need more drive space etc, but that's about it.

[Hardcore-fs]

ok its got it..... dropping it to 32768 fixed it....
that's the strangest S** iv'e seen....

[P1] Table 1 took 413.396 sec

i'm also thinking that the drive for phase 1 , it writes 127 of 300MB, that is stuff that could be held in ram.
then after calc do a separate thread out to the disk drive.

I'm just wondering if it actually needs a memaloc for this buffer on osx, instead of a hard build.

[madMAx43v3r]

Ah maybe you got a stack-overflow from it...
Also check latest master, it's working now, as far as I can see. See readme.

[Hardcore-fs]

and onto the next one....
[P1] Table 1 took 413.396 sec
[P1] Table 2 took 1263.02 sec, found 4295006334 matches
Segmentation fault: 11

don't see how... there is 96gb of ram, i think maybe the implementation in the kernel for fread is bad

[Hardcore-fs]

ok same reason but in
/include/chia/DiskTable.h
perhaps there can be a global def. for buffers?

[madMAx43v3r]

yeah that's a good idea, also I should allocate the buffers on heap, not on the stack.

[madMAx43v3r]

check src/settings.cpp for easy tuning of those params, in the future also via command line