raise LateEnd() on some scrypt coins

[squiggie]

I've been trying to implement p2pool for some alt coins and lately I've been getting some error messages on several new alt coins that I'm not able to resolve. I'm fairly confident I've been able to input the proper values for the network.py files, but I just can't figure out what the error is referring to.

I was able to track it down these functions.

def _unpack(self, data, ignore_trailing=False):
        obj, (data2, pos) = self.read((data, 0))

        assert data2 is data

        if pos != len(data) and not ignore_trailing:
            raise LateEnd()

        return obj

 def unpack(self, data, ignore_trailing=False):
        obj = self._unpack(data, ignore_trailing)

        if p2pool.DEBUG:
            packed = self._pack(obj)
            good = data.startswith(packed) if ignore_trailing else data == packed
            if not good:
                raise AssertionError()

        return obj

If ignore_trailing is set to True, the error goes away. I'm assuming that is because the root of the issue is pos is always 1 character larger then data. I just don't know enough about these functions to know what is going on here. I'm not that versed in python.

I changed these values to True and testing this overnight with a coin and although we were able to find blocks, none of them made it into the block chain and no payouts were made. So I assume changing that is messing with the block somehow.

Any advice would be appreciated in how to alter this and make it work for other alt coins.

[squiggie]

Sorry, here is the error that is thrown.

 Testing bitcoind RPC connection to 'http://127.0.0.1:5555/' with username 'user'...
2013-12-30 00:24:16.668394 > Error getting work from bitcoind:
2013-12-30 00:24:16.668474 > Traceback (most recent call last):
2013-12-30 00:24:16.668509 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
2013-12-30 00:24:16.668538 >     current.result = callback(current.result, *args, **kw)
2013-12-30 00:24:16.668566 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1095, in gotResult
2013-12-30 00:24:16.668598 >     _inlineCallbacks(r, g, deferred)
2013-12-30 00:24:16.668626 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
2013-12-30 00:24:16.668675 >     result = result.throwExceptionIntoGenerator(g)
2013-12-30 00:24:16.668704 >   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
2013-12-30 00:24:16.668732 >     return g.throw(self.type, self.value, self.tb)
2013-12-30 00:24:16.668759 > --- <exception caught here> ---
2013-12-30 00:24:16.668785 >   File "/home/x/p2pool-eac/p2pool/util/deferral.py", line 41, in f
2013-12-30 00:24:16.668812 >     result = yield func(*args, **kwargs)
2013-12-30 00:24:16.668838 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
2013-12-30 00:24:16.668866 >     result = g.send(result)
2013-12-30 00:24:16.668892 >   File "/home/x/p2pool-eac/p2pool/bitcoin/helper.py", line 49, in getwork
2013-12-30 00:24:16.668919 >     transactions=map(bitcoin_data.tx_type.unpack, packed_transactions),
2013-12-30 00:24:16.668945 >   File "/home/x/p2pool-eac/p2pool/util/pack.py", line 63, in unpack
2013-12-30 00:24:16.668972 >     obj = self._unpack(data, ignore_trailing)
2013-12-30 00:24:16.668998 >   File "/home/x/p2pool-eac/p2pool/util/pack.py", line 47, in _unpack
2013-12-30 00:24:16.669024 >     raise LateEnd()
2013-12-30 00:24:16.669050 > p2pool.util.pack.LateEnd:

[forrestv]

This probably means that p2pool.bitcoin.data.tx_type, the definition of the binary layout of a transaction, doesn't match the layout that this *coin is using. Check the changes the *coin made for something like that.

Can you post all your changes somewhere so I can try to replicate?

[squiggie]

Sure. I made detailed post here. https://bitcointalk.org/index.php?topic=214512.new#new

That details the network.py values I used as well as the functions I altered to ignore_trailing. That's really the only changes I made. I can't remember if I used your source or the rav3n source https://github.com/Rav3nPL/p2pool-rav but I don't think there are very many differences except the network.py values.

If you want, I can zip and and post the p2pool folder i'm running and testing with on my vm. Would that be helpful?

[squiggie]

Any idea where the tx type definition is in the coins source usually? I'm struggling to find it, but I could troubleshoot that if I knew in the coin source where it was.

[litecoinsblog]

I'm also looking into adding more scrypt coins to p2pool. If anyone else wants to work on this together to expand the number of ALT coins supported by p2pools then feel free to PM me or contact SEOCrow on skype. It will benefit all of us if we can determine whats causing these issues with specific ALT Coins.

[squiggie]

It does appear the the structure of that call has been modified a little bit. It appears that a std::string strTxComment; has been added to it. However I am unsure of how to alter the pack.tx_type to work with this addition. Any help would be appreciated.

[squiggie]

Here are the network.py values I altered. There were not other code changes except for the above mentions ignore_trailing=True that I mentioned.

 earthcoin=math.Object(
        PARENT=networks.nets['earthcoin'],
        SHARE_PERIOD=10, # seconds
        CHAIN_LENGTH=3*60*60//5, # shares
        REAL_CHAIN_LENGTH=3*60*60//5, # shares
        TARGET_LOOKBEHIND=120, # shares
        SPREAD=30, # blocks
        IDENTIFIER='e137d5b8c6923410'.decode('hex'),
        PREFIX='7218c1a53ef629b0'.decode('hex'),
        P2P_PORT=9338,
        MIN_TARGET=0,
        MAX_TARGET=2**256//2**20 - 1,
        PERSIST=False,
        WORKER_PORT=19330,
        BOOTSTRAP_ADDRS=''.split(' '),
        ANNOUNCE_CHANNEL='#spool',
        VERSION_CHECK=lambda v: True,
    ),

 earthcoin=math.Object(
        P2P_PREFIX='c0dbf1fd'.decode('hex'),
        P2P_PORT=15677,
        ADDRESS_VERSION=93,
        RPC_PORT=15678,
        RPC_CHECK=defer.inlineCallbacks(lambda bitcoind: defer.returnValue(
            'earthcoinaddress' in (yield bitcoind.rpc_help()) and
            not (yield bitcoind.rpc_getinfo())['testnet']
        )),
        SUBSIDY_FUNC=lambda height: 10000*100000000 >> (height + 1)//525600,
        POW_FUNC=lambda data: pack.IntType(256).unpack(__import__('ltc_scrypt').getPoWHash(data)),
        BLOCK_PERIOD=60, # s
        SYMBOL='EAC',
        CONF_FILE_FUNC=lambda: os.path.join(os.path.join(os.environ['APPDATA'], 'earthcoin') if platform.system() == 'Windows' else os.path.expanduser('~/Library/Application Support/earthcoin/') if platform.system() == 'Darwin' else os.path.expanduser('~/.earthcoin'), 'earthcoin.conf'),
        BLOCK_EXPLORER_URL_PREFIX='http://earthchain.info/block/',
        ADDRESS_EXPLORER_URL_PREFIX='http://earthchain.info/address/',
        TX_EXPLORER_URL_PREFIX='http://earthchain.info/tx/',
        SANE_TARGET_RANGE=(2**256//1000000000 - 1, 2**256//1000 - 1),
        DUMB_SCRYPT_DIFF=2**16,
        DUST_THRESHOLD=1e8,
    ),

[sgphk]

Squiggie, I see you have managed to get the pool up and running after all :-)
What was the trick?

[squiggie]

No, i haven't been able to make this work with some alt coins that throw this error. It works with some alt coins but not others. The examples I have above for earthcoin are a good example.

[sgphk]

I'm trying to prepare in advance for a new coin based on eac but am running into the exact same issue. I figure it's because of the transaction comment deal...I spent hours on this yesterday and kept ending up back here at this thread over and over. Tried multiple releases, clean installs, different versions, commenting out lines, and now I'm just getting frustrated. Other coins that have transaction comment like Florin (FLO) never got added into P2Pool so I'm guessing this is a bigger issue than either of us think.

What needs to be done is have p2pool recognize that the comment line exists and either do something with it or hardcode it to ignore it outright. Preferably I'd like to use the comment line if possible to say the payout came from which pool.

[hyacin75]

I've tried to modify the tx_type format in p2pool/bitcoin/data.py a few different ways and I keep running in to errors. The issue seems to be that sometimes there is a tx comment, and sometimes there isn't, and I have no clue how to handle this. I have plenty of scripting experience, but very little programming experience, and pretty much no python experience. Any pointers on how to handle packing a field that may or may not be there would be GREATLY appreciated. I've tried playing with this "PossiblyNoneType" a few different ways as I'm guessing that's how you handle it, but I'm still getting errors no matter what I do.

The most recent error I'm getting when I use ('txcomment', pack.PossiblyNoneType("", pack.VarStrType())) or ('txcomment', pack.PossiblyNoneType(0, pack.IntType(256))) is -

2014-03-10 12:08:20.391968 > Unhandled Error
2014-03-10 12:08:20.392043 > Traceback (most recent call last):
2014-03-10 12:08:20.392093 >   File "/home/rob/p2pool/p2pool/main.py", line 578, in run
2014-03-10 12:08:20.392138 >     reactor.run()
2014-03-10 12:08:20.392180 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2014-03-10 12:08:20.392248 >     self.mainLoop()
2014-03-10 12:08:20.392295 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2014-03-10 12:08:20.392338 >     self.runUntilCurrent()
2014-03-10 12:08:20.392379 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2014-03-10 12:08:20.392421 >     call.func(*call.args, **call.kw)
2014-03-10 12:08:20.392461 > --- <exception caught here> ---
2014-03-10 12:08:20.392502 >   File "/home/rob/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2014-03-10 12:08:20.392544 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2014-03-10 12:08:20.392600 >   File "/home/rob/p2pool/p2pool/bitcoin/worker_interface.py", line 129, in get_work
2014-03-10 12:08:20.392643 >     x, handler = self._inner.get_work(*args)
2014-03-10 12:08:20.392685 >   File "/home/rob/p2pool/p2pool/work.py", line 294, in get_work
2014-03-10 12:08:20.392726 >     packed_gentx = bitcoin_data.tx_type.pack(gentx)
2014-03-10 12:08:20.392767 >   File "/home/rob/p2pool/p2pool/util/pack.py", line 74, in pack
2014-03-10 12:08:20.392807 >     data = self._pack(obj)
2014-03-10 12:08:20.392847 >   File "/home/rob/p2pool/p2pool/util/pack.py", line 52, in _pack
2014-03-10 12:08:20.392888 >     f = self.write(None, obj)
2014-03-10 12:08:20.392928 >   File "/home/rob/p2pool/p2pool/util/pack.py", line 301, in write
2014-03-10 12:08:20.392969 >     file = type_.write(file, item[key])
2014-03-10 12:08:20.393010 >   File "/home/rob/p2pool/p2pool/util/pack.py", line 315, in write
2014-03-10 12:08:20.393050 >     raise ValueError('none_value used')
2014-03-10 12:08:20.393090 > exceptions.ValueError: none_value used