Bootstrap.dat does not work beyond a certain block height

[switzer]

Per my conversation with @oyvkva on Slack the bootstrap.dat file does not help with block hight much higher than 200K.

As we build wallets for more consumer-y folks, an installation process that takes days will surely limit the number of wallets that will be installed.

[GroundRod]

Greetings Switzer,

You are new to me on the dev team. Although 'new' would be everyone whom came along after '15, so that is ages.......ago in Cryptoland.

This issue from 2018 I think can be closed, stuck syncs happened, almost always fixable given each persons configuration issue that was causing the problem. Yet your comment gave me pause:

As we build wallets for more consumer-y folks, an installation process that takes days will surely limit the number of wallets that will be installed.

First the answer to this question seems clear now. YES
or if you meant it as a statement and not a question, I would agree it limits the number of installs..... 'bigtime'

...it goes to the heart of an important matter to me though and these last few years I've been thinking about. Could we build our trusted blockchain right into the latest wallet release? A solution for today, signed by multiple devs whom trust is already implied. We need instant installs to support the new users coming on board, sub one hour to-the-tip syncs a must.

Along with a comparably small blockchain size on disk Anoncoin imo still has allot of potential to return as a major development effort in the CrytpoVerse...

You decide...

GR

[mikalv]

@GroundRod great work with the PID stuff, however we discovered a very big issue once we compiled for OSX. We spent weeks trying to solve this, and it turned out we hit an undefined behaviour in C++. Meaning, it's up to the compiler developer to decide what to do, it's not a standard on it.

Please check out;
commit 7c79f2f
commit d5d6c9b

The issue was that std::sort was used instead of std::stable_sort.
https://en.cppreference.com/w/cpp/algorithm/sort
https://en.cppreference.com/w/cpp/algorithm/stable_sort

Cause, in PID often blocks had the same timestamp, and what sort lacks, and stable_sort do is that it sorts a range of elements while preserving order between equal elements. So because sort was used, OSX won't sync and have the same blockchain as Linux/Windows. We tried patching this with using a lot of checkpoints since it wasn't really a good solution otherwise in code terms.

[mikalv]

I think the above is related to exactly this issue due to how bootstrap.dat works, which was probably made on a Linux - but won't sync after a block on OSX.

[GroundRod]

Mikalv

GOOD TO HEAR FROM YOU!

Thanks much for the feedback which cost so many of your hours and work by others to debug and fix. Will study this history in detail, and not forget the lesson learned.

Amazing another 'undefined behavior in C++' problem encountered...

... yet as you might have guessed I knew nothing of stable_sort, until now. Where it failed on OsX would have been rare, no wonder this was a hard bug to find. Will try to review the commit range this weekend. Plus had to think about it for awhile, but ya I get it Mikalv. Why its related to the bootstrap failure.

[mikalv]

Yea no worries, we did some heavy research and discovered something like that if blocks had the same timestamp and it got more than 10 of them, clang and gcc started doing the sorting a bit differently while I mean to recall that clang did some odd ordering for some reason. Anyhow, not worth digging into - let's rather focus on the focus on the future. I should have for example stealth addresses ready to be merged into production ANC. Same goes for BIP32 addresses.

[GroundRod]

Mikalv

Thanks for all of that. Thanks to for your kind words on my development of the PID controller for Anoncoin. As you know, its a revolutionary new approach to retarget blockchain mining difficulty. My history with and passion for its development goes way back and I'm so proud of the team here for getting it working right and on the mainnet while I was away.

Ya agree, we can save that discussion for another day, right now focus on the future. Seems there are many matters need tending too, yet I'm trying to keep it simple for starters.

Getting setup once again to build production code on Debian Stretch is mostly working, just as August hit, I'm upgrading to Debian Buster on several machines...will soon rebuild the toolchains and see if I can fix some issues and upgrade the libraries to newer versions.

My plan is to use Buster's version of GCC for building executables. Will leave to others for builds with CLang and/or for OsX. The Windows 64bit builds I can test here on a Dell I5 notebook running Win8.1. Hopefully the Setup.exe installs nicely on Win10 too, will look into Busters Windows Setup.exe generator tool, confirm and/or re-write that build process, if needed.

Using Docker Compose will be new for me, and I might add, its exciting to finally have a system like docker invented and working so well! Docker appears to be a great addition to our build stack, but kinda learning about it from scratch these last few months.

Mikalv let me know if these development tool plans suit our needs & the rest of the team or if I need to make other arrangements.

Besides trying to fix some of the build problems, most likely some of my 1st commits will be to upgrade our tool chain to the latest versions that I see Bitcoin & Litecoin using.

Knowing you will be focused on Stealth issues and BIP32 addressing for integration into the production code sounds like an awesome plan. Getting a new set of executables built for everyone is my goal. It will be more than nice to have Hierarchical Deterministic Wallet addresses added to the feature set.

Onward...
GR

[mikalv]

We did the first step towards stealth and BIP32 at the last fork, cause we need all nodes to accept OP_RETURN. Also worth notice is the "depends" directory in the source tree, it has makefiles for building both the daemon and it's dependencies for all platforms, however I think it's sadly not much used.

For the BIP32 addresses we needed to change some unused parameters in chainparams.cpp to fit our wanted prefix of such addresses, so in theory it's already fully possible to send ANCs to a address of that kind. But I haven't sitten down and actually tested that yet, but I'm gonna do it within a couple of days if my planning goes as wanted.

Another thing to expect is that I'm gonna throw out the current i2psam implementation in favor of https://github.com/i2p/i2psam which also supports new signature types and not only the old DSA. I want to get us over to Ed25519 which the linked library already does.

[GroundRod]

Wow this is all very important details I need to know moving forward. Thankyou Mikalv.

Familiar with the old OP_RETURN code issues and possible solutions, will take a look so I understand what implementation you guys went with and how it was coded.

Once setup here & building targets from source, participating in & documenting BIP32 tests would be a great 1st step towards helping everyone use the feature. Also, I will study in detail these 'depends' change histories and run all the new depends build targets that I can. That aspect of things could effect my operational procedure for releases allot.

Ya agree if you want to push for a new standard in i2psam lets do it now, I'm not yet familiar with the new Ed25519 signature type, but will learn & as it sounds like its backwards compatible I see no need to delay.

GR

For those that don't know:
Depends was/is an important subsystem ToolChain builder, added to Bitcoin in the v9.6-v1.0 time frame I think. So anyway, long ago I ported it to Anoncoin & started modifying to our needs, finally after couple years I understood how it works & importantly too, 'why' its needed and 'when' to use it. Think I've forgotten allot of that detail, but pretty sure it will come back to me quickly when I get to working on the code again.

To those left wondering:
My goals are pretty simple at the moment. No substantial code changes, except what Mikalv is proposing. For my part-> Get us a cleaner & nicer build process with fresh new release images in tow.