JSON Web API

[dternyak]

I'd like to create some some Web Apps based on the extremely performant and feature-rich back-end of this project. This will be much easier to do if there is a web-facing JSON API to interact with.

Ideally, the JSON API would expose the following endpoints (to start):

/api/v1/mempool
/api/v1/search?blkhash='blah'/?txhash/etc. Ideally all the supported search types of the main site.
/api/v1/transactions?page=1
/api/v1/transactions?page=2
/api/v1/tx/<tx_hash>

I'll work to develop the spec of the JSON response itself for each endpoint, but sane defaults will also be fine. Basically whatever is already being returned to the traditional back-end rendered template handlerbar/mustache engine would be great.

I'm not sure how much work this is, or how realistic it is to even ask for this, but I'm hoping that it can be fairly easily integrated given the existing architecture.

However, I also understand that this project has a focus on darknet support, which is not especially compatible with the type of service I am proposing. Perhaps a fork of this project is a better proposal?

Eager to hear your thoughts. This is probably an ambitious ask, so please let me know if I am overreaching.

[moneroexamples]

Any feature requests and feedback are welcome.

At the moment not sure if it would be good to add them in the explorer, or make it a separate project. Open Monero project's aim is all about making REST JSON service for monero, so maybe it would be easier to add it there. But then maybe explorer is better for this in this case, as your api endpoints are more specific to the explorer's functionality. Don't know yet, guess it depends what data would you like to see in the json response.

If you have an example of a specific json response, I can try adding it to the explorer first, if its not too complicated, and see how it works.

But like I wrote in the earlier issue, before I start working "full time" on something new, such as json or cmake builds, I need to finish other things. Thus, not sure when I could add full api support, except adding one or two endpoints for quick tests at the moment, if they would not require lots of changes to existing codebase.

[dternyak]

The endpoint are indeed essentially identical to what the block explorer currently has, but I agree OpenMonero is centered around REST JSON services.

Here's what I came up with for the spec for the first two endpoints:

{
    "api/v1/tx/<tx_hash>": {
        "timestamp": '',
        "timestampUTC": '',
        "block": '',
        "confirmations": '',
        "outputs": [{
            "stealAddress": '',
            "amount": ''
        }]
    },
    "/api/v1/transactions?page=2&limit=25": [
        {
            "height": '',
            "age": '',
            "size": '',
            "txHash": '',
            "fees": '',
            "outputs": '',
            "mixin": '',
            "txSize": ''
        }
    ]
}

It would be nice if the transactions API let you set the limit like I describe, maybe with a cap at 25. I can see myself only needing 10 usually, but sometimes wanting more or less, so it would be nice to have a dynamic limit.

The pagination might tricky if you start including limits, so feel free to not do limits if it starts becoming annoying. More of a nice-to-have.

[moneroexamples]

Thanks. I will try to come up with something next week, and will let you know if something has happened in this regard.

[moneroexamples]

JSON api was added. Information are in README in this branch:

• https://github.com/moneroexamples/onion-monero-blockchain-explorer/tree/try_lru_cache2

If you want you can test it, see what is there, what is missing, etc. Any feedback welcomed as usual.

[dternyak]

Wow!!! Going to give this a spin now.

[dternyak]

Looks amazing so far. I've already gotten started on prototyping my web app now that these API's are in place.

Thanks for doing this.

[dternyak]

Could you enable cross origin requests (CORS)? Maybe this link helps... ipkn/crow#201

[dternyak]

This is what I'm working on in case you're curious :)

Trying to make a really UX friendly UI for people who are okay with having Javascript.

[dternyak]

@moneroexamples I'm noticing that tx_fee (at least for mempool, probably for tx too) is some variable amount of digits.

Could it always be 12 units, with 0's all the way out until it hits the 12 digits?

I even converted your python get_money script (from here https://moneroexamples.github.io/python-json-rpc/) to javascript, but I get weird results.

function get_money(amount) {
    var CRYPTONOTE_DISPLAY_DECIMAL_POINT, idx, s;
    /*decode cryptonote amount format to user friendly format. Hope its correct.

    Based on C++ code:
    https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_core/cryptonote_format_utils.cpp#L751
    */
    CRYPTONOTE_DISPLAY_DECIMAL_POINT = 12;
    s = amount;
    if ((s.length < (CRYPTONOTE_DISPLAY_DECIMAL_POINT + 1))) {
        s = (("0" * ((CRYPTONOTE_DISPLAY_DECIMAL_POINT + 1) - s.length)) + s);
    }
    idx = (s.length - CRYPTONOTE_DISPLAY_DECIMAL_POINT);
    s = ((s.slice(0, idx) + ".") + s.slice(idx));
    return s;
}

[moneroexamples]

Hi

I added the header for cors:

r.add_header("Access-Control-Allow-Origin", "*");
r.add_header("Access-Control-Allow-Headers", "Content-Type");

Fees are just represented as unsigned integer of base 1e12. So divide them by 1e12 to get them in user "float" format.

p.s.
Looks cool what you are doing. But if you pull data from my server, remember its development only server. So I would not use it for anything serious, as it often goes offline, restarts etc.

[dternyak]

Duh! That did the trick.

Thanks for disabling CORS - yeah I'll make sure to run my own instance once this goes to prod.

Here's another screenshot - the main page design is essentially done, going to clean up the code and then work on various detail views.

[moneroexamples]

Looks good. As a side not you can also pull some info from monero deamon itself, if something is not available through the explorer. Also maybe can add something extra functionality. For example, some statistics about network. Currently there is this http://monerostats.com/ but i think its dead now. There is also this http://moneroblocks.info/stats but its rather basic.

p.s.
If you have any requests for what can be added to json api or change, please let me know.

[dternyak]

I'm getting close to a public release

Here's where I am right now: http://monerochain-166203.appspot.com/

I'll have an ability to switch "nodes" - basically whatever domain the API is hosted on.

There's some mobile weirdness that I have to fix, and some other misc tweaks, but its mostly done.

Tell me what you think!

[moneroexamples]

Looks awesome. The only comment I can have at the moment is that on front page it shows last 25 txs, but there is no button to show older ones, like going to next page back in the blockchain. Not sure this is intentional or not? Also probably in tx page, you would need to rename the fields to be more user friendly. For example, payment_id8 is internal name for encrypted payment id, those made when using integrated addresses. End users may not know what payment_id8 is. But they might now what encrypted payment id means.

[dternyak]

@moneroexamples Yup - pagination is being worked on. And yeah, some of those fields should probably be changed. I'm just using whatever the API returns right now

[dternyak]

@moneroexamples Could you add the # of total pages in the response for /api/transactions?page=2&limit=10? :)

[moneroexamples]

Just added it. Its in total_page_no variable.

As a side note, api/transaction has been extended: public keys of mixins have been added to key images in api/transaction calls.

Let me know if something else is needed or something is buggy:-)

[dternyak]

Thank you so much! One thing - I previously suggested a pattern like `/api/v1/<blah>`, in part because changes to existing fields can have unintended affects. Adding the public keys of the mixins (making mixins value a dictionary instead of just a flat value) actually broke my table due since it expected flat data. Obviously I'm using your dev environment for my dev so I shouldn't assume any stability there, but it would be nice if the api had versioning so that cool updates like this could exist without changing the structure of the data for the current version.

…

On Sun, May 7, 2017 at 1:22 AM, moneroexamples ***@***.***> wrote: Just added it. Its in total_page_no variable. As a side note, api/transaction has been extended: public keys of mixins have been added to key images in api/transaction calls. Let me know if something else is needed or something is buggy:-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#55 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHf02WoLo8Bz7g36MEF2EddsSZ96tdAPks5r3WM2gaJpZM4M-iTK> .

[moneroexamples]

I remember and sorry for breaking things. The explorer itself does not have any versioning or tags yet. Its more a rolling release, like arch linux, and sometimes things break unfortunately.

But I think that in future will start tagging it. Was thinking to be in sync with monero releases. So probably when monero will have next release, I will tag the explorer as well.

At the moment I dont plan making more changes to existing json, unless there are some bugs or feature requests.

Again sorry for breaking your tx page.

[dternyak]

No worries! Thanks again for adding the total_pages so quickly.

And a tagged, versioned release sounds good!

[dternyak]

@moneroexamples Check it out - added paging. Last thing to add is ability to choose back-end "node" (REST API host) and then I think its good for a 1.0 release!

http://monerochain-166203.appspot.com/

[dternyak]

I do have one additional request actually - mempool is beginning to get quite large (200 currently) and I forsee it getting bigger as monero grows in popularity.

Could pagination (and page limiting) be included in the mempool api?

[moneroexamples]

400 now and 16MB. woow :-)

Pagination for mempool is problematic, because it changes very fast. So whatever page number you get in a first request, in a second request, few minutes later, it might be incorrect.

That's said I think a good start would be to add option, e.g. limit=20 to show top 20 mempool txs. Something what explorer is doing right now. Might also experiment with doing full pagination anyway, just to see how would it work, if have time.

But I cant add this today. Maybe tomorrow.

[moneroexamples]

Just the heads up. Mempool api will break. I'm adding limits and pages, so it will be similar to api/transactions api. Its not yet live on development server, but changes are being made in this branch: https://github.com/moneroexamples/onion-monero-blockchain-explorer/tree/json_mempool_add_limit

[dternyak]

Cool - so you did decide to go with paginating the mempool?

Can I ask that you add a total_pages similar to transactions if you do?

And one last thing - could you also add a total_tx_in_mempool that lists the total # of all txs in the mempool? That will be helpful to display on the UI.

[moneroexamples]

I will add pagination as well, just to check how it work. Results will be interesting, as mempool changes rapidly.
JSON response will be similar to transactions. So total_page_no and txs_no (i.e., total number of tx in memool) will be there as well.

The new mempool is already live for tests. Its not merged yet to master, as need more testing, you can have a look already how it shapes up.

# all txs in mempool
http://139.162.32.245:8081/api/mempool

# newest 5 txs in mempool
http://139.162.32.245:8081/api/mempool?limit=5

# newest 5 in 3th page (first page is page=0)
http://139.162.32.245:8081/api/mempool?limit=2&page=2

As a side note. If you use chrome or chromium, I use this extension to nicely parse the json output from those url:
https://chrome.google.com/webstore/detail/json-formatter/bcjindcccaagfpapjjmafapmmgkkhgoa?utm_source=chrome-app-launcher-info-dialog

Maybe it will be helpful, if you dont use something similar already.

[dternyak]

I do use that extension already! I like it.

Could you let me know when the new mempool is up on your dev server? Doesn't seem to be on the dev server right now (139.162.32.245) actually seems to be down.

Excited to implement this final thing and then move to a stable release!

[dternyak]

@moneroexamples Great idea! Actually the fact that your dev server is http solves the issue of safari / edge, so I may even default to that unless you say not to.

And yes, good idea for adding testnet support. I'm going to do that as well, and then put up a banner saying testnet when its enabled.

[moneroexamples]

The existent explorer json api wont be changing anymore I think. But I will add one or two new calls probably. Having said that the explorer will be going off line from time to time as I work on it. Currently, adding some network info to index pages as per this issue:

#59

So having the dev server as default is not good idea in my view, unless you can detect that its off online and switch to alternative one quickly.

I think your project is very nice. once it grows and more nodes will join, can have unique set of stats and information constructed using the aggregate of these nodes.

[dternyak]

Interesting... @suhz A bit of a silly ask, but any chance you could host a non-https explorer on another subdomain? The reason I need this is because certain browsers (edge, safari), don't like it when you make cross-domain cross http-https requests, and glitch out.

Let me know if thats something you're willing to do :)

[notsuhz]

@dternyak Sorry, I do not think I can do that because xmr.my have HSTS preloaded, including subdomains. it will still redirect to https even if that subdomain is not exist.

Another way to do this is to host the api on new top level domain. Or, maybe you can create new subdomain (eg: api.monerochain.com), and CNAME it to my server (node.xmr.my). I can configure my server to serve the API for your domain.

[moneroexamples]

@dternyak

Maybe changing headers for json would help? Currently the headers are:

add_header("Access-Control-Allow-Origin", "*");
add_header("Access-Control-Allow-Headers", "Content-Type");
add_header("Content-Type", "application/json");

If you know any better or more permissive ones, I can change it.

[dternyak]

@moneroexamples - I don't think its a backend header thing, I think its a browser thing.

@suhz - having CNAME point to your server from my subdomain would be awesome. I'm not super familiar with DNS and how that works - is it as simple as adding a CNAME record with setting host to api and then target to node.xmr.my?

I tried the above, but it didn't work, so I probably did something wrong.

[notsuhz]

@dternyak I see that you've already pointed it to my server. It didn't work earlier because I haven't yet configure it for your domain.

It works now, try http://api.monerochain.com/api/mempool

EDIT
Just a note that, I cannot guarantee that it will have 100% uptime. And I might limit and/or cache incoming request in future if it receive a lot of traffic.

[dternyak]

@suhz Thank you! And I definitely understand about the limiting. How has the traffic been so far? I've purposefully added no analytics whatsoever so I really have no way of knowing.

[notsuhz]

@dternyak I don't log either. But I can see how much bandwidth it consume each day.

[moneroexamples]

fyi. just added version to api.

https://github.com/moneroexamples/onion-monero-blockchain-explorer/tree/update_to_current_monero#apiversion

[moneroexamples]

@suhz

Not sure you know but https://moneroexplorer.com/ node seems down. Also, just merged huge PR into master in preparation for the next monero release. Just in case you want to use current master, latest development version of monero is needed.

[notsuhz]

@moneroexamples thanks, actually its still syncing. I've accidently deleted whole .bitmonero folder few days ago, where I wanted to actually just delete the one for testnet. silly me.

[moneroexamples]

@suhz

No problem. The merged PR has lots of changes, including templates. This is this PR: #75

Most imporantly, mempool has been revamp. Now it is its own, separate thread, that periodically checks the mempool. Up until now, each http request resulted in mempool check, which could be very intensive for a server. Hopefully now, mempool processing will be better.

[notsuhz]

@moneroexamples I was unable to compile the latest explorer, it is related to ssl something.

I unistall openssl that comes with ubuntu 16.04, compile & install openssl 1.1, and still didn't help.

Here is the error - https://pastebin.com/raw/5Lnc8wzm

[moneroexamples]

From the pastebin, it seems that one reason could be that explorer did not find libcncrypto.a

• FindMonero Xmr_CNCRYPTO_LIBRARIES Xmr_CNCRYPTO_LIBRARY-NOTFOUND

Are you sure you are running recent monero development version? This is new library, not present in 10.3.1, from what I recall. Also, whenever you run cmake .. again, its often good idea to remove all files build before that. Sometime it will contain some cashed info, which may result in some things not found or not updated.

[notsuhz]

okay, I pull latest monero master (commit 48f96c18dfa8d2f5c86dc7c6c538b0b003992200) fresh into new folder and built it.

compiling explorer, also in fresh build folder, Xmr_CNCRYPTO_LIBRARIES issue seems fixed. but there is still error related to SSL which I do not understand.

Please have a look - https://pastebin.com/raw/qYAvth9P

[moneroexamples]

On ubuntu I use this dependencies for ssl: libssl-dev libcurl4-openssl-dev. Did you try with them? It seems that crow is blowing up.

[notsuhz]

I have that installed already

root@lax:# apt install libssl-dev libcurl4-openssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libcurl4-openssl-dev is already the newest version (7.47.0-1ubuntu2.2).
libssl-dev is already the newest version (1.1.0f-2~ubuntu16.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 62 not upgraded.

[moneroexamples]

I think I know what is happening. Same issue was reported here microsoft/cpprestsdk#132 and was identified as bug of boost with openssl 1.1.

You are running libssl-dev is already the newest version (1.1.0f-2~ubuntu16.04.1+deb.sury.org+1)., while I have libssl-dev is already the newest version (1.0.2g-1ubuntu4.8). So you running openssl 1.1, which boost has problems with. Can you use earlier openssl version?.

[notsuhz]

note sure what happen with my VPS, I ended up reinstalling it with fresh ubuntu. now it works. Thanks :)

[moneroexamples]

good it works now:-)

[dternyak]

@moneroexamples Any chance you could double check to see you enabled CORS? I would like to piggy back off of xmrchain.net (as @Gingeropolous has previously seemed ok with it), but the server is responding with:

... has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource

[dternyak]

In the meantime, I could proxy requests via my own backend, but this somewhat defeats the purpose of an entirely client side blockchain explorer that works with any onion-monero-blockchain-explorer compatible JSON API.

[moneroexamples]

@dternyak

I have these headers set for api:

        add_header("Access-Control-Allow-Origin", "*");
        add_header("Access-Control-Allow-Headers", "Content-Type");
        add_header("Content-Type", "application/json");

If other values or new heards are needed, I would be grateful for info how and what to set.

I dont know if xmrchain is using the onion explorer version with these headers. Can you check using this explorer: https://moneroexplorer.com/ This one for sure should be using latest version and have these headers.

[dternyak]

@moneroexamples Using MoneroExplorer does indeed fix my problem -- XmrChain.net must not have been on the latest version (and missing those headers as a result).

http://monerochain.com is fully functioning once again! Thank you for your help.

[moneroexamples]

@dternyak

Cool it works. Thanks for the info.

[dternyak]

@moneroexamples I think this issue has served it's purpose ;)

I'll be closing this one now, thanks again on your hard fork maintaining this and other projects.

[moneroexamples]

No problem, and thanks for help with the api.