New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add websocket support for a wallet/account/address? #3388

Closed
doodzik opened this Issue Dec 11, 2013 · 21 comments

Comments

Projects
None yet
4 participants
@doodzik

doodzik commented Dec 11, 2013

would it be desirable to implement the support of the websocket protocol into the client,
so that you can subscribe to new transactions for a wallet or accounts or specific addresses?
I'm not familiar with the source code, so i don't know if it is possible, but if it is or considered a good idea i would volunteer to implement this feature.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 11, 2013

Member

In principle, getting notified is already possible through the ***notify options:

  -blocknotify=<cmd>     Execute command when the best block changes (%s in cmd is replaced by block hash)
  -walletnotify=<cmd>    Execute command when a wallet transaction changes (%s in cmd is replaced by TxID)
  -alertnotify=<cmd>     Execute command when a relevant alert is received or we see a really long fork (%s in cmd is replaced by message)

Websocket support for asynchronous notification of events without having to provide scripts would be nice though.

Could this use the same port as the normal RPC mechanism but with a different URI?

Member

laanwj commented Dec 11, 2013

In principle, getting notified is already possible through the ***notify options:

  -blocknotify=<cmd>     Execute command when the best block changes (%s in cmd is replaced by block hash)
  -walletnotify=<cmd>    Execute command when a wallet transaction changes (%s in cmd is replaced by TxID)
  -alertnotify=<cmd>     Execute command when a relevant alert is received or we see a really long fork (%s in cmd is replaced by message)

Websocket support for asynchronous notification of events without having to provide scripts would be nice though.

Could this use the same port as the normal RPC mechanism but with a different URI?

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 11, 2013

i'm not very happy with the ***notify options.

I thought that you could access the ws mechanism by the same Authority and add a path or scheme/urn of "ws".
for example:

192.0.2.16:8080/ws
or
ws://192.0.2.16:8080

by sending a message you could subscribe to which notifications you want, like with the blockchain.info websockets api, but with more flexibility.

the messages could look something like this:

// authentification
{"op": "auth", "user": "username", "password":"password"}
// sets default confirmation other then six
{"op": "minconfirm", "minconfirm": "0"}
// adds transactions of the given address with min confirmation of 6 to the subscriptions list
{"op": "add", "minconfirm": "6", "addr":"$btcAddr"}
// adds transactions of all addresses of the given acc with default minconfirm to the subscriptions list
{"op": "add", "acc":"$btcAcc"}
// adds transactions of the whole wallet with default minconfirm to the subscription list
{"op": "add"}
// removes transactions of the given address from the subscriptions list
{"op": "remove", "addr":"$btcAddr"}
// removes transactions of the given acc from the subscriptions list
{"op": "remove", "acc":"$btcAcc"}
// resets transactions the subscriptions list
{"op": "remove"}

// if you send the same message only with a different minconfirm the old one will be overwritten

I think the best way to implement ws support would be to use the ***notify options as a fundament and build the ws as a layer on top of it. what do you think?

doodzik commented Dec 11, 2013

i'm not very happy with the ***notify options.

I thought that you could access the ws mechanism by the same Authority and add a path or scheme/urn of "ws".
for example:

192.0.2.16:8080/ws
or
ws://192.0.2.16:8080

by sending a message you could subscribe to which notifications you want, like with the blockchain.info websockets api, but with more flexibility.

the messages could look something like this:

// authentification
{"op": "auth", "user": "username", "password":"password"}
// sets default confirmation other then six
{"op": "minconfirm", "minconfirm": "0"}
// adds transactions of the given address with min confirmation of 6 to the subscriptions list
{"op": "add", "minconfirm": "6", "addr":"$btcAddr"}
// adds transactions of all addresses of the given acc with default minconfirm to the subscriptions list
{"op": "add", "acc":"$btcAcc"}
// adds transactions of the whole wallet with default minconfirm to the subscription list
{"op": "add"}
// removes transactions of the given address from the subscriptions list
{"op": "remove", "addr":"$btcAddr"}
// removes transactions of the given acc from the subscriptions list
{"op": "remove", "acc":"$btcAcc"}
// resets transactions the subscriptions list
{"op": "remove"}

// if you send the same message only with a different minconfirm the old one will be overwritten

I think the best way to implement ws support would be to use the ***notify options as a fundament and build the ws as a layer on top of it. what do you think?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 11, 2013

Member

The idea looks very sane to me.

Note that there are wallet (walletnotify) and general client notifications (such as blocknotify, alertnotify). It helps to keep those separate conceptually.

I'm not entirely sure the connection should be stateful, this brings a lot of micro-management. Is it really necessary to change subscriptions on the fly?

Try to make it as simple as possible, at least at first, so that we don't introduce too much new code to the core.

Member

laanwj commented Dec 11, 2013

The idea looks very sane to me.

Note that there are wallet (walletnotify) and general client notifications (such as blocknotify, alertnotify). It helps to keep those separate conceptually.

I'm not entirely sure the connection should be stateful, this brings a lot of micro-management. Is it really necessary to change subscriptions on the fly?

Try to make it as simple as possible, at least at first, so that we don't introduce too much new code to the core.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 12, 2013

Sorry I meant walletnotify, but I read a little in the source code and I think the ws support doesn't need to be dependent of walletnotify. A WsBroadcast method could be included somewhere at ./src/wallet.cpp:504 and it should work without any problems.

i think it could be helpful if you can just alter the subscription List without restarting the bitcoin/bitcoind process and it shouldn't cost much networking or processing. it would add a bit of flexibility, which some devs need(myself included) and it would definately separate it from the ***notify options.

I provided a concept which would use the minconfirm of wallatenotify. Please dont hate me for the provided pseudocode, i had my first experience in c/c++ about 3 hours ago. I hope that the concept is understandable.

using Alchemy;
using Alchemy.Classes;

static void cWs(string[] args, config *)
{
    var cWsServer = new WebSocketServer(config.port, config.IpAddress) {
        OnReceive = OnReceive
    };
    // you should set ws = 1 in the bitcoin.config file if you want to start the WsServer.
    // if you don't do it in the command line, osx devs can get notifications without building their own bitcoind  
    if (config.ws == true)
    {
        cWsServer.Start();
    }
}

public static void WsBroadcast(object transactionData, WsContext context)
{
    foreach (var u in WsUsers.Keys)
    {
        if (transactionData.addr in u.context.addreses)
        {
            u.Context.Send(JsonConvert.SerializeObject(transactionData));   
        }
    }
}

private static void OnReceive(WsContext context)
{
    try
    {
        var json = context.DataFrame.ToString();
        dynamic obj = JsonConvert.DeserializeObject(json);

        switch ((string)obj.cmd)
        {
            case "auth":
                Auth(obj.user, obj.pass);
                break;
            case "add":
                AddOrRemove(false, obj)
                break;
            case "remove":
                AddOrRemove(true, obj)
                break;
        }
    }
    catch (Exception e) // Bad JSON! For shame.
    {
        var r = new Response {Type = ResponseType.Error, Data = new {e.Message}};

        context.Send(JsonConvert.SerializeObject(r));
    }
}

private static void Auth(string name, string pass, WsContext context)
{
    var u = WsUsers.Keys.Where(o => o.Context.ClientAddress == context.ClientAddress).Single();
    if (ValidateUser(name, pass)) {
        context.Allowed = true;
        WsUsers[u] = name;
    }
    else
    {
       context.Allowed = false;
       SendError("incorrect user or password", context);
    }
}

// AddOrRemove is more or less combined with javascript so dont wonder
private static void AddOrRemove(bool remove, object obj, WsContext context)
{
    array.arrMethod = (remove) ? array.remove : array.push;
    if (obj.addr.GetType() == typeof(object))
    {
        foreach (var addr in obj.addr)
        {
            context.addr.arrMethod(addr);
        }
    }
    else if (obj.addr.GetType() == typeof(string))
    {
        context.addr.arrMethod(obj.addr);
    }
    else if (obj.acc.GetType() == typeof(string))
    {
        context.addr.arrMethod(cWallet.getAddressesByAccount(obj.acc));
    }
    else
    {
        context.addr.arrMethode(cWallet.getAddresses());   
    }
}

// and then in src/wallet.cpp:504 include WsBroadcast(transactionData)

doodzik commented Dec 12, 2013

Sorry I meant walletnotify, but I read a little in the source code and I think the ws support doesn't need to be dependent of walletnotify. A WsBroadcast method could be included somewhere at ./src/wallet.cpp:504 and it should work without any problems.

i think it could be helpful if you can just alter the subscription List without restarting the bitcoin/bitcoind process and it shouldn't cost much networking or processing. it would add a bit of flexibility, which some devs need(myself included) and it would definately separate it from the ***notify options.

I provided a concept which would use the minconfirm of wallatenotify. Please dont hate me for the provided pseudocode, i had my first experience in c/c++ about 3 hours ago. I hope that the concept is understandable.

using Alchemy;
using Alchemy.Classes;

static void cWs(string[] args, config *)
{
    var cWsServer = new WebSocketServer(config.port, config.IpAddress) {
        OnReceive = OnReceive
    };
    // you should set ws = 1 in the bitcoin.config file if you want to start the WsServer.
    // if you don't do it in the command line, osx devs can get notifications without building their own bitcoind  
    if (config.ws == true)
    {
        cWsServer.Start();
    }
}

public static void WsBroadcast(object transactionData, WsContext context)
{
    foreach (var u in WsUsers.Keys)
    {
        if (transactionData.addr in u.context.addreses)
        {
            u.Context.Send(JsonConvert.SerializeObject(transactionData));   
        }
    }
}

private static void OnReceive(WsContext context)
{
    try
    {
        var json = context.DataFrame.ToString();
        dynamic obj = JsonConvert.DeserializeObject(json);

        switch ((string)obj.cmd)
        {
            case "auth":
                Auth(obj.user, obj.pass);
                break;
            case "add":
                AddOrRemove(false, obj)
                break;
            case "remove":
                AddOrRemove(true, obj)
                break;
        }
    }
    catch (Exception e) // Bad JSON! For shame.
    {
        var r = new Response {Type = ResponseType.Error, Data = new {e.Message}};

        context.Send(JsonConvert.SerializeObject(r));
    }
}

private static void Auth(string name, string pass, WsContext context)
{
    var u = WsUsers.Keys.Where(o => o.Context.ClientAddress == context.ClientAddress).Single();
    if (ValidateUser(name, pass)) {
        context.Allowed = true;
        WsUsers[u] = name;
    }
    else
    {
       context.Allowed = false;
       SendError("incorrect user or password", context);
    }
}

// AddOrRemove is more or less combined with javascript so dont wonder
private static void AddOrRemove(bool remove, object obj, WsContext context)
{
    array.arrMethod = (remove) ? array.remove : array.push;
    if (obj.addr.GetType() == typeof(object))
    {
        foreach (var addr in obj.addr)
        {
            context.addr.arrMethod(addr);
        }
    }
    else if (obj.addr.GetType() == typeof(string))
    {
        context.addr.arrMethod(obj.addr);
    }
    else if (obj.acc.GetType() == typeof(string))
    {
        context.addr.arrMethod(cWallet.getAddressesByAccount(obj.acc));
    }
    else
    {
        context.addr.arrMethode(cWallet.getAddresses());   
    }
}

// and then in src/wallet.cpp:504 include WsBroadcast(transactionData)
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 12, 2013

Member

Right

  • Having to restart bitcoind is undesirable, I agree to that point.
  • I also agree that specifying minconf can be useful

However: what I mean is that the categories should be broad categories like

  • "incoming transaction"
  • "a transaction has N confirmations" (where N is configurable)
  • "initial sync completed"
  • "new block/best chain"
  • "an alert"
  • "a node connected"
  • "a node disconnected".. etc. Broad enough to be specified at connection time, or after that.

Per address is way too fine grained. The interface should be as much of a passive listener as possible. Just listen for the events, deliver the messages to the client and let them sort it out.

Don't perform the fine-grained filtering for the client. Having all kinds of subscribe/unsubscribe traffic after the initial connection handshake is too much micromanagement / over-design.

Remember that 99% of clients will be interested in everything that happens in a wallet. If not, you should probably be running multiple wallets in the first place.

Member

laanwj commented Dec 12, 2013

Right

  • Having to restart bitcoind is undesirable, I agree to that point.
  • I also agree that specifying minconf can be useful

However: what I mean is that the categories should be broad categories like

  • "incoming transaction"
  • "a transaction has N confirmations" (where N is configurable)
  • "initial sync completed"
  • "new block/best chain"
  • "an alert"
  • "a node connected"
  • "a node disconnected".. etc. Broad enough to be specified at connection time, or after that.

Per address is way too fine grained. The interface should be as much of a passive listener as possible. Just listen for the events, deliver the messages to the client and let them sort it out.

Don't perform the fine-grained filtering for the client. Having all kinds of subscribe/unsubscribe traffic after the initial connection handshake is too much micromanagement / over-design.

Remember that 99% of clients will be interested in everything that happens in a wallet. If not, you should probably be running multiple wallets in the first place.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 12, 2013

right you have a point there.

how would you handle the subscription on different events?
i see three possibilities:

  1. emit everything that happens
  2. emit only on a specific event after it is defined. {"user": "username", "pass": "password", "sub": "transactions"}
  3. have a different path for each event ws://192.0.2.16:8080/transactions

Im in favor of the second option.

the ws support could fix the problem with the too many opened threads when using walletnotify.

doodzik commented Dec 12, 2013

right you have a point there.

how would you handle the subscription on different events?
i see three possibilities:

  1. emit everything that happens
  2. emit only on a specific event after it is defined. {"user": "username", "pass": "password", "sub": "transactions"}
  3. have a different path for each event ws://192.0.2.16:8080/transactions

Im in favor of the second option.

the ws support could fix the problem with the too many opened threads when using walletnotify.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 12, 2013

Member

Although neatest, the drawback of (3) is that it allows to subscribe only to one kind of event for a connection. So you need to make multiple connections to wait for multiple kinds of events.

So you need to be able to specify a list of event types to subscribe to. The URI syntax does work that well with that.

Member

laanwj commented Dec 12, 2013

Although neatest, the drawback of (3) is that it allows to subscribe only to one kind of event for a connection. So you need to make multiple connections to wait for multiple kinds of events.

So you need to be able to specify a list of event types to subscribe to. The URI syntax does work that well with that.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 12, 2013

something like this?

ws://192.0.2.16:8080/transactions/block/initialSync

hmh, i don't know. I Would prefer to send a message and the ws server would respond to It, as it is done by a ws client most of the time. You would need to send an authentification anyhow, so this way you wouldn't need to do it twice and even if you include the authentification in the URI it wouldn't be pretty.

{"user": "username", "pass": "password", "sub": ["transactions", "block", "initialSync"]}

Also it would allow to add new functionality easily. For example transactions with a specified min confirm.

{"user": "username", "pass": "password", "sub": [{"transactions": {"minconfirm": 10}}, "block"]}
{"user": "username", "pass": "password", "sub": [{"transactions": 10}, "block"]}
{"user": "username", "pass": "password", "sub": {"transactions": 10}}
{"user": "username", "pass": "password", "sub": "transactions", "minconfirm": 10}

doodzik commented Dec 12, 2013

something like this?

ws://192.0.2.16:8080/transactions/block/initialSync

hmh, i don't know. I Would prefer to send a message and the ws server would respond to It, as it is done by a ws client most of the time. You would need to send an authentification anyhow, so this way you wouldn't need to do it twice and even if you include the authentification in the URI it wouldn't be pretty.

{"user": "username", "pass": "password", "sub": ["transactions", "block", "initialSync"]}

Also it would allow to add new functionality easily. For example transactions with a specified min confirm.

{"user": "username", "pass": "password", "sub": [{"transactions": {"minconfirm": 10}}, "block"]}
{"user": "username", "pass": "password", "sub": [{"transactions": 10}, "block"]}
{"user": "username", "pass": "password", "sub": {"transactions": 10}}
{"user": "username", "pass": "password", "sub": "transactions", "minconfirm": 10}
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 12, 2013

Member

Yes, that looks good.

Member

laanwj commented Dec 12, 2013

Yes, that looks good.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 12, 2013

the URI syntax or JSON?

doodzik commented Dec 12, 2013

the URI syntax or JSON?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 12, 2013

Member

The JSON. The URI syntax is strange, as / indicates a hierarchy.

Member

laanwj commented Dec 12, 2013

The JSON. The URI syntax is strange, as / indicates a hierarchy.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 12, 2013

ok then. Im going to write something in the next couple of days.
I'll keep you updated

doodzik commented Dec 12, 2013

ok then. Im going to write something in the next couple of days.
I'll keep you updated

@doodzik

This comment has been minimized.

Show comment
Hide comment

doodzik commented Dec 15, 2013

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 15, 2013

Member

Ideally we'd like to avoid needing even more dependencies.

In any case:

Member

laanwj commented Dec 15, 2013

Ideally we'd like to avoid needing even more dependencies.

In any case:

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Dec 23, 2013

I have finished the websocket script for broadcasting. It isn't dependent on the bitcoin source code and it works as a module.
I am going to write unit tests for it after Christmas, but before I start I have a few questions:

  • How can I access the data in the config file?
  • Should I implement my code differently because of #3440?
  • Which unit testisting framework should I use?
  • How can I get more information about a transaction/block with a transaction/block hash? I want to emit something like the blockchain.info websocket api

doodzik commented Dec 23, 2013

I have finished the websocket script for broadcasting. It isn't dependent on the bitcoin source code and it works as a module.
I am going to write unit tests for it after Christmas, but before I start I have a few questions:

  • How can I access the data in the config file?
  • Should I implement my code differently because of #3440?
  • Which unit testisting framework should I use?
  • How can I get more information about a transaction/block with a transaction/block hash? I want to emit something like the blockchain.info websocket api
@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Jan 19, 2014

any updates?

doodzik commented Jan 19, 2014

any updates?

@sandalsoft

This comment has been minimized.

Show comment
Hide comment
@sandalsoft

sandalsoft Jan 31, 2014

This looks like the continuation of #1897. I'm interested in pushing this feature forward. I'm happy to help writing code or testing.

@doodzik - I'm new here so I could be wrong, but at first glance the config file parsing is done using boost libs and is found here: https://github.com/bitcoin/bitcoin/blob/master/src/util.cpp#L1046.

sandalsoft commented Jan 31, 2014

This looks like the continuation of #1897. I'm interested in pushing this feature forward. I'm happy to help writing code or testing.

@doodzik - I'm new here so I could be wrong, but at first glance the config file parsing is done using boost libs and is found here: https://github.com/bitcoin/bitcoin/blob/master/src/util.cpp#L1046.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Feb 2, 2014

@sandalsoft -
Thank you
i have to accomplish a couple of deadlines over the next weeks so i wont be able to do it fast.
But I will try to work a bit on it on the next weekend.

doodzik commented Feb 2, 2014

@sandalsoft -
Thank you
i have to accomplish a couple of deadlines over the next weeks so i wont be able to do it fast.
But I will try to work a bit on it on the next weekend.

@sandalsoft

This comment has been minimized.

Show comment
Hide comment
@sandalsoft

sandalsoft Feb 2, 2014

@doodzik No worries. I'll hack on the repo in the meantime.

sandalsoft commented Feb 2, 2014

@doodzik No worries. I'll hack on the repo in the meantime.

@doodzik

This comment has been minimized.

Show comment
Hide comment
@doodzik

doodzik Apr 30, 2014

Im closing this issue, because it would be more secure to setup an own service which isnt connected to the core(e.g. https://github.com/bitpay/bitcorie).
And also zeromq would be better than websockets for this use case.

doodzik commented Apr 30, 2014

Im closing this issue, because it would be more secure to setup an own service which isnt connected to the core(e.g. https://github.com/bitpay/bitcorie).
And also zeromq would be better than websockets for this use case.

@leegod

This comment has been minimized.

Show comment
Hide comment
@leegod

leegod May 1, 2018

What is zeromq and how to monitoring all transactions incoming to specific wallet? (specific bitcoin daemon run on server) I want to implement exchange's bitcoin wallet.

leegod commented May 1, 2018

What is zeromq and how to monitoring all transactions incoming to specific wallet? (specific bitcoin daemon run on server) I want to implement exchange's bitcoin wallet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment