Backport bitcoin perutxo

[codablock]

Continuation of #1691

This PR contains the actual per-utxo patch (bitcoin#10195) and fixes which came later in Bitcoin.

With this PR, Dash will do a one-time db-upgrade to the new schema on first start. While this happens, dash-qt will show the progress on the loading screen. This progress is currently badly positioned. If it's ok for you all, I'll fix this in a later PR.

[codablock]

Hmm, I got compilation errors in most of the Travis jobs. On my laptop it compiles just fine.

This is most likely due to gcc version differences. Does anyone know which gcc version is used in Travis builds? @flare maybe?

[schinzelh]

@codablock I'll check against gitian as well, Travis-CI throws this

coins.h:119:14: error:   initializing argument 1 of ‘CCoinsCacheEntry::CCoinsCacheEntry(Coin&&)’
     explicit CCoinsCacheEntry(Coin&& coin_) : coin(std::move(coin_)), flags(0) {}
              ^

[schinzelh]

gitian confirms compilation error:

In file included from /usr/include/x86_64-linux-gnu/c++/4.8/32/bits/c++allocator.h:33:0,
                 from /usr/include/c++/4.8/bits/allocator.h:46,
                 from /usr/include/c++/4.8/string:41,
                 from /usr/include/c++/4.8/random:41,
                 from /usr/include/c++/4.8/bits/stl_algo.h:65,
                 from /usr/include/c++/4.8/algorithm:62,
                 from ./serialize.h:11,
                 from ./amount.h:9,
                 from primitives/transaction.h:9,
                 from compressor.h:9,
                 from coins.h:9,
                 from coins.cpp:5:
/usr/include/c++/4.8/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator< <template-parameter-1-1> >::construct(_Up*, _Args&& ...) [with _Up = CCoinsCacheEntry; _Args = {Coin&}; _Tp = boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> >]’:
/usr/include/c++/4.8/bits/alloc_traits.h:254:4:   required from ‘static typename std::enable_if<std::allocator_traits<_Alloc>::__construct_helper<_Tp, _Args>::value, void>::type std::allocator_traits<_Alloc>::_S_construct(_Alloc&, _Tp*, _Args&& ...) [with _Tp = CCoinsCacheEntry; _Args = {Coin&}; _Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; typename std::enable_if<std::allocator_traits<_Alloc>::__construct_helper<_Tp, _Args>::value, void>::type = void]’
/usr/include/c++/4.8/bits/alloc_traits.h:393:57:   required from ‘static decltype (_S_construct(__a, __p, (forward<_Args>)(std::allocator_traits::construct::__args)...)) std::allocator_traits<_Alloc>::construct(_Alloc&, _Tp*, _Args&& ...) [with _Tp = CCoinsCacheEntry; _Args = {Coin&}; _Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; decltype (_S_construct(__a, __p, (forward<_Args>)(std::allocator_traits::construct::__args)...)) = <type error>]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/allocate.hpp:917:51:   required from ‘void boost::unordered::detail::func::call_construct(Alloc&, T*, Args&& ...) [with Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; T = CCoinsCacheEntry; Args = {Coin&}]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/allocate.hpp:1057:4:   required from ‘void boost::unordered::detail::func::construct_from_tuple(Alloc&, T*, const std::tuple<A0>&) [with Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; T = CCoinsCacheEntry; A0 = Coin&&]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/allocate.hpp:1124:35:   required from ‘typename boost::enable_if<boost::unordered::detail::func::use_piecewise<A0>, void>::type boost::unordered::detail::func::construct_from_args(Alloc&, std::pair<_Tp1, _Tp2>*, A0&&, A1&&, A2&&) [with Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; A = const COutPoint; B = CCoinsCacheEntry; A0 = const std::piecewise_construct_t&; A1 = std::tuple<const COutPoint&>; A2 = std::tuple<Coin&&>; typename boost::enable_if<boost::unordered::detail::func::use_piecewise<A0>, void>::type = void]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/allocate.hpp:1338:48:   required from ‘typename boost::unordered::detail::allocator_traits<Alloc>::pointer boost::unordered::detail::func::construct_node_from_args(Alloc&, Args&& ...) [with Alloc = std::allocator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; Args = {const std::piecewise_construct_t&, std::tuple<const COutPoint&>, std::tuple<Coin&&>}; typename boost::unordered::detail::allocator_traits<Alloc>::pointer = boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> >*]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/unique.hpp:424:80:   required from ‘boost::unordered::detail::table_impl<Types>::emplace_return boost::unordered::detail::table_impl<Types>::emplace_impl(const key_type&, Args&& ...) [with Args = {const std::piecewise_construct_t&, std::tuple<const COutPoint&>, std::tuple<Coin&&>}; Types = boost::unordered::detail::map<std::allocator<std::pair<const COutPoint, CCoinsCacheEntry> >, COutPoint, CCoinsCacheEntry, SaltedOutpointHasher, std::equal_to<COutPoint> >; boost::unordered::detail::table_impl<Types>::emplace_return = std::pair<boost::unordered::iterator_detail::iterator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >, bool>; typename boost::unordered::detail::table<Types>::iterator = boost::unordered::iterator_detail::iterator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >; boost::unordered::detail::table_impl<Types>::key_type = COutPoint]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/detail/unique.hpp:360:48:   required from ‘boost::unordered::detail::table_impl<Types>::emplace_return boost::unordered::detail::table_impl<Types>::emplace(Args&& ...) [with Args = {const std::piecewise_construct_t&, std::tuple<const COutPoint&>, std::tuple<Coin&&>}; Types = boost::unordered::detail::map<std::allocator<std::pair<const COutPoint, CCoinsCacheEntry> >, COutPoint, CCoinsCacheEntry, SaltedOutpointHasher, std::equal_to<COutPoint> >; boost::unordered::detail::table_impl<Types>::emplace_return = std::pair<boost::unordered::iterator_detail::iterator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >, bool>; typename boost::unordered::detail::table<Types>::iterator = boost::unordered::iterator_detail::iterator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >]’
/home/ubuntu/build/dash/depends/i686-pc-linux-gnu/share/../include/boost/unordered/unordered_map.hpp:269:64:   required from ‘std::pair<typename boost::unordered::detail::map<A, K, T, H, P>::table::iterator, bool> boost::unordered::unordered_map<K, T, H, P, A>::emplace(Args&& ...) [with Args = {const std::piecewise_construct_t&, std::tuple<const COutPoint&>, std::tuple<Coin&&>}; K = COutPoint; T = CCoinsCacheEntry; H = SaltedOutpointHasher; P = std::equal_to<COutPoint>; A = std::allocator<std::pair<const COutPoint, CCoinsCacheEntry> >; typename boost::unordered::detail::map<A, K, T, H, P>::table::iterator = boost::unordered::iterator_detail::iterator<boost::unordered::detail::ptr_node<std::pair<const COutPoint, CCoinsCacheEntry> > >]’
coins.cpp:48:146:   required from here
/usr/include/c++/4.8/ext/new_allocator.h:120:4: error: cannot bind ‘Coin’ lvalue to ‘Coin&&’
  { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
    ^
In file included from coins.cpp:5:0:
coins.h:119:14: error:   initializing argument 1 of ‘CCoinsCacheEntry::CCoinsCacheEntry(Coin&&)’
     explicit CCoinsCacheEntry(Coin&& coin_) : coin(std::move(coin_)), flags(0) {}
              ^

[schinzelh]

Sidenote: gitian is using gcc-4.8.4 on Ubuntu Trusty, same as Bitcoin 0.15 - so the issue must be somewhere in the code :)

ubuntu@gitian:~$ gcc --version
+ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[codablock]

Setting up gitian now, because even if I install gcc-4.8 locally it's still not reproducable

[codablock]

Ok, builds are passing now. It was not a difference in gcc version, but a difference in boost versions and the use of boost::unordered_map vs. std::unordered_map. At the same time, I skipped backporting bitcoin#10249 as I planned to have C++11 related changes as part of my other backportings. I added this PR now, which changes boost::unordered_map to std::unordered_map, which also fixes the issue I had with compilation.

[OlegGirko]

I've spent some time yesterday trying to find the cause of this problem, but I was too late to report my findings.

The problem is indeed with using emplace() method of boost::unordered_map with 3 arguments std::piecewise_construct, tuple of arguments for key, tuple of arguments for value.
This is a simplified test case to check for this problem:

#include <utility>
#include <tuple>
#include <boost/unordered_map.hpp>

class C {
    int a_;
public:
    explicit C(int &&a): a_(std::move(a)) {}
};

int main() {
    boost::unordered_map<int, C> c_map;
    c_map.emplace(std::piecewise_construct,
                  std::forward_as_tuple(1),
                  std::forward_as_tuple(2));
    return 0;
}

Compilation fails in both Ubuntu with GCC 4.8.4 and Fedora with GCC 7.2.1.
Changing boost::unordered_map to std::unordered_map in example above fixes the problem.

Deep inside implementation of boost::unordered_map rvalue reference is lost when using std::get()` function template to retrieve tuple element. Here is a test case:

include <utility>
#include <tuple>

class C {
    int a_;
public:
    explicit C(int &&a): a_(std::move(a)) {}
};

int main() {
    std::tuple<int&&, int&&> t = std::forward_as_tuple(1, 2);
    C c(std::get<0>(t));
    return 0;
}

It's interesting to note that std::unordered_map uses std::get() as well, but in combination with std::forward(), so it doesn't lose rvalue reference. If you wrap std::get<0>(t) inside std::forward<int&&>() in the example above, compilation problem is fixed.

[OlegGirko]

Although this particular problem is solved by using std::unordered_map instead of boost::unordered_map, I decided to check whether the problem with boost::unordered_map still exists in Boost and found that it was fixed in Boost 1.65: it has emplace() method that forwards arguments to std::pair's constructor, and leaves special handling of std::piecewise_construct only for the case when two remaining arguments have boost::tuple type.

It's interesting to note that there is also no problem with emplacing using std::piecewise_construct in earlier versions of Boost (tested with Boost 1.54). This is because earlier versions have no special handling for std::piecewise_construct yet, so they just pass 3 arguments (std::piecewise_construct and 2 tuples) straight to std::pair's constructor that is already C++11-compatible and handles these arguments correctly.

[codablock]

Now Travis isn't even starting the build with error "An error occurred while generating the build script."
@schinzelh Can you check whats wrong?

[schinzelh]

@codablock Travis is down —> https://www.traviscistatus.com

[codablock]

Ah ok, next time I'll check that by myself :D

About the failing tests I had after fixing the build issue: These were caused by an erroneous merge resolution I did in dash-tx.cpp. I removed too much code which was surrounded by SegWit changes. This is fixed now and will hopefully give green builds after Travis is up again.

[codablock]

Builds are finally green. After fixing the previous test failures, I got the next one which only happened in Travis but not locally. "make check" was failing with out-of-bounds access to a vector, without telling me which one it was. After some Travis debugging with some log outputs and increased BOOST_TEST_LOG_LEVEL, it turned out that some stream read with size 0 was performed on an empty vector, which is fine normally, but crashes with stl debugging enabled. Adding bitcoin#10250 to this PR fixed the issue.

I'm repushing a clean version now (omitting the debug code), after this gives green Travis builds, we can finally merge.

[UdjinM6]

Nice job, thanks! 👍

utACK