add websocket support for a wallet/account/address?

[doodzik]

would it be desirable to implement the support of the websocket protocol into the client,
so that you can subscribe to new transactions for a wallet or accounts or specific addresses?
I'm not familiar with the source code, so i don't know if it is possible, but if it is or considered a good idea i would volunteer to implement this feature.

[laanwj]

In principle, getting notified is already possible through the ***notify options:

  -blocknotify=<cmd>     Execute command when the best block changes (%s in cmd is replaced by block hash)
  -walletnotify=<cmd>    Execute command when a wallet transaction changes (%s in cmd is replaced by TxID)
  -alertnotify=<cmd>     Execute command when a relevant alert is received or we see a really long fork (%s in cmd is replaced by message)

Websocket support for asynchronous notification of events without having to provide scripts would be nice though.

Could this use the same port as the normal RPC mechanism but with a different URI?

[doodzik]

i'm not very happy with the ***notify options.

I thought that you could access the ws mechanism by the same Authority and add a path or scheme/urn of "ws".
for example:

192.0.2.16:8080/ws
or
ws://192.0.2.16:8080

by sending a message you could subscribe to which notifications you want, like with the blockchain.info websockets api, but with more flexibility.

the messages could look something like this:

// authentification
{"op": "auth", "user": "username", "password":"password"}
// sets default confirmation other then six
{"op": "minconfirm", "minconfirm": "0"}
// adds transactions of the given address with min confirmation of 6 to the subscriptions list
{"op": "add", "minconfirm": "6", "addr":"$btcAddr"}
// adds transactions of all addresses of the given acc with default minconfirm to the subscriptions list
{"op": "add", "acc":"$btcAcc"}
// adds transactions of the whole wallet with default minconfirm to the subscription list
{"op": "add"}
// removes transactions of the given address from the subscriptions list
{"op": "remove", "addr":"$btcAddr"}
// removes transactions of the given acc from the subscriptions list
{"op": "remove", "acc":"$btcAcc"}
// resets transactions the subscriptions list
{"op": "remove"}

// if you send the same message only with a different minconfirm the old one will be overwritten

I think the best way to implement ws support would be to use the ***notify options as a fundament and build the ws as a layer on top of it. what do you think?

[laanwj]

The idea looks very sane to me.

Note that there are wallet (walletnotify) and general client notifications (such as blocknotify, alertnotify). It helps to keep those separate conceptually.

I'm not entirely sure the connection should be stateful, this brings a lot of micro-management. Is it really necessary to change subscriptions on the fly?

Try to make it as simple as possible, at least at first, so that we don't introduce too much new code to the core.

[doodzik]

Sorry I meant walletnotify, but I read a little in the source code and I think the ws support doesn't need to be dependent of walletnotify. A WsBroadcast method could be included somewhere at ./src/wallet.cpp:504 and it should work without any problems.

i think it could be helpful if you can just alter the subscription List without restarting the bitcoin/bitcoind process and it shouldn't cost much networking or processing. it would add a bit of flexibility, which some devs need(myself included) and it would definately separate it from the ***notify options.

I provided a concept which would use the minconfirm of wallatenotify. Please dont hate me for the provided pseudocode, i had my first experience in c/c++ about 3 hours ago. I hope that the concept is understandable.

using Alchemy;
using Alchemy.Classes;

static void cWs(string[] args, config *)
{
    var cWsServer = new WebSocketServer(config.port, config.IpAddress) {
        OnReceive = OnReceive
    };
    // you should set ws = 1 in the bitcoin.config file if you want to start the WsServer.
    // if you don't do it in the command line, osx devs can get notifications without building their own bitcoind  
    if (config.ws == true)
    {
        cWsServer.Start();
    }
}

public static void WsBroadcast(object transactionData, WsContext context)
{
    foreach (var u in WsUsers.Keys)
    {
        if (transactionData.addr in u.context.addreses)
        {
            u.Context.Send(JsonConvert.SerializeObject(transactionData));   
        }
    }
}

private static void OnReceive(WsContext context)
{
    try
    {
        var json = context.DataFrame.ToString();
        dynamic obj = JsonConvert.DeserializeObject(json);

        switch ((string)obj.cmd)
        {
            case "auth":
                Auth(obj.user, obj.pass);
                break;
            case "add":
                AddOrRemove(false, obj)
                break;
            case "remove":
                AddOrRemove(true, obj)
                break;
        }
    }
    catch (Exception e) // Bad JSON! For shame.
    {
        var r = new Response {Type = ResponseType.Error, Data = new {e.Message}};

        context.Send(JsonConvert.SerializeObject(r));
    }
}

private static void Auth(string name, string pass, WsContext context)
{
    var u = WsUsers.Keys.Where(o => o.Context.ClientAddress == context.ClientAddress).Single();
    if (ValidateUser(name, pass)) {
        context.Allowed = true;
        WsUsers[u] = name;
    }
    else
    {
       context.Allowed = false;
       SendError("incorrect user or password", context);
    }
}

// AddOrRemove is more or less combined with javascript so dont wonder
private static void AddOrRemove(bool remove, object obj, WsContext context)
{
    array.arrMethod = (remove) ? array.remove : array.push;
    if (obj.addr.GetType() == typeof(object))
    {
        foreach (var addr in obj.addr)
        {
            context.addr.arrMethod(addr);
        }
    }
    else if (obj.addr.GetType() == typeof(string))
    {
        context.addr.arrMethod(obj.addr);
    }
    else if (obj.acc.GetType() == typeof(string))
    {
        context.addr.arrMethod(cWallet.getAddressesByAccount(obj.acc));
    }
    else
    {
        context.addr.arrMethode(cWallet.getAddresses());   
    }
}

// and then in src/wallet.cpp:504 include WsBroadcast(transactionData)

[laanwj]

Right

• Having to restart bitcoind is undesirable, I agree to that point.
• I also agree that specifying minconf can be useful

However: what I mean is that the categories should be broad categories like

• "incoming transaction"
• "a transaction has N confirmations" (where N is configurable)
• "initial sync completed"
• "new block/best chain"
• "an alert"
• "a node connected"
• "a node disconnected".. etc. Broad enough to be specified at connection time, or after that.

Per address is way too fine grained. The interface should be as much of a passive listener as possible. Just listen for the events, deliver the messages to the client and let them sort it out.

Don't perform the fine-grained filtering for the client. Having all kinds of subscribe/unsubscribe traffic after the initial connection handshake is too much micromanagement / over-design.

Remember that 99% of clients will be interested in everything that happens in a wallet. If not, you should probably be running multiple wallets in the first place.

[doodzik]

right you have a point there.

how would you handle the subscription on different events?
i see three possibilities:

1. emit everything that happens
2. emit only on a specific event after it is defined. {"user": "username", "pass": "password", "sub": "transactions"}
3. have a different path for each event ws://192.0.2.16:8080/transactions

Im in favor of the second option.

the ws support could fix the problem with the too many opened threads when using walletnotify.

[laanwj]

Although neatest, the drawback of (3) is that it allows to subscribe only to one kind of event for a connection. So you need to make multiple connections to wait for multiple kinds of events.

So you need to be able to specify a list of event types to subscribe to. The URI syntax does work that well with that.

[doodzik]

something like this?

ws://192.0.2.16:8080/transactions/block/initialSync

hmh, i don't know. I Would prefer to send a message and the ws server would respond to It, as it is done by a ws client most of the time. You would need to send an authentification anyhow, so this way you wouldn't need to do it twice and even if you include the authentification in the URI it wouldn't be pretty.

{"user": "username", "pass": "password", "sub": ["transactions", "block", "initialSync"]}

Also it would allow to add new functionality easily. For example transactions with a specified min confirm.

{"user": "username", "pass": "password", "sub": [{"transactions": {"minconfirm": 10}}, "block"]}
{"user": "username", "pass": "password", "sub": [{"transactions": 10}, "block"]}
{"user": "username", "pass": "password", "sub": {"transactions": 10}}
{"user": "username", "pass": "password", "sub": "transactions", "minconfirm": 10}

[laanwj]

Yes, that looks good.

[doodzik]

the URI syntax or JSON?

[laanwj]

The JSON. The URI syntax is strange, as / indicates a hierarchy.

[doodzik]

ok then. Im going to write something in the next couple of days.
I'll keep you updated

[doodzik]

can i use these libraries:
https://bitbucket.org/tunnuz/json
https://github.com/zaphoyd/websocketpp

[laanwj]

Ideally we'd like to avoid needing even more dependencies.

In any case:

• https://bitbucket.org/tunnuz/json: no, use the json-spirit that is already built-in for JSON parsing/formatting
• https://github.com/zaphoyd/websocketpp: yes, if it is possible to plug it into the current RPC code. I suppose this makes sense as you'd need some websocket implementation.

[doodzik]

I have finished the websocket script for broadcasting. It isn't dependent on the bitcoin source code and it works as a module.
I am going to write unit tests for it after Christmas, but before I start I have a few questions:

• How can I access the data in the config file?
• Should I implement my code differently because of [RFC] Post-0.9 modularization #3440?
• Which unit testisting framework should I use?
• How can I get more information about a transaction/block with a transaction/block hash? I want to emit something like the blockchain.info websocket api

[doodzik]

any updates?

[sandalsoft]

This looks like the continuation of #1897. I'm interested in pushing this feature forward. I'm happy to help writing code or testing.

@doodzik - I'm new here so I could be wrong, but at first glance the config file parsing is done using boost libs and is found here: https://github.com/bitcoin/bitcoin/blob/master/src/util.cpp#L1046.

[doodzik]

@sandalsoft -
Thank you
i have to accomplish a couple of deadlines over the next weeks so i wont be able to do it fast.
But I will try to work a bit on it on the next weekend.

[sandalsoft]

@doodzik No worries. I'll hack on the repo in the meantime.

[doodzik]

Im closing this issue, because it would be more secure to setup an own service which isnt connected to the core(e.g. https://github.com/bitpay/bitcorie).
And also zeromq would be better than websockets for this use case.

[leegod]

What is zeromq and how to monitoring all transactions incoming to specific wallet? (specific bitcoin daemon run on server) I want to implement exchange's bitcoin wallet.