Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

web: split up into REST api server and a js client #316

Closed
TomK32 opened this issue Jan 8, 2016 · 18 comments

Comments

Projects
None yet
4 participants
@TomK32
Copy link
Contributor

commented Jan 8, 2016

I had a look through old issues concerning hledger-web and little progress seems to be made, and my limited knowledge of haskell keeps me from doing more for it. Which is why I'd like to propose a very radical step:

  • Cut down the yesod app to serve as a REST API
  • Move all logic and templates into a javascript app (i don't care wether we use Angular, Backbone or Ember)

This move would bring the web-clients language to the browsers native language javascript, allowing easier modifications for the plenty numbers of javascript programmers out there. Logic and view templates could be separated far better than they are right now. For remaining server, a wider and purer REST api would allow inter-machine JSON requests, imagine a POS or webshop sending details about your sales directly into your bookkeeping!

For reference, there's an unfinished(?) project to do the same for ledger using python and AngularJS: https://groups.google.com/forum/#!msg/ledger-cli/2gRLc9kCPCc/Cw_4KG1dJqwJ -- source -- docs

and another old project, for ledger: https://github.com/slashdotdash/node-ledger-web


Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Jan 9, 2016

Yes, this has been on the wishlist for a long time. I'm leaning towards this approach: create a new hledger-api package containing

  1. a series of simple example client-side web apps
  2. the simplest possible web API server (I'm trying to avoid misusing "REST" here) that makes them work. I'd give servant a try.

The examples should always work, and drive the evolution of the server.

simonmichael added a commit that referenced this issue Jan 9, 2016

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Jan 10, 2016

We now have a minimal API server and client-side example. Please build more examples/apps and let me know what API methods you need added to the server (pull requests for that also welcome).

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 11, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 11, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 11, 2016

@TomK32

This comment has been minimized.

Copy link
Contributor Author

commented Jan 12, 2016

I'd prever /accounts to return a tree like this:

[
  {"name": "Assets",
    "children": [
      {"name": "TownBank",
       "children": [
          {"name": "Checking"}
       ]
      },
      { "name": "CreditCard"}
    ]
  },
  {"name": "Liabilities"},
]
@TomK32

This comment has been minimized.

Copy link
Contributor Author

commented Jan 12, 2016

The next resource i need is the balances for an account.
GET /accounts/Expenses:Food/balances
to return

[
  { "name": "USD", "amount": 200 },
  { "name": "", "amount": 440 }
]

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 12, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 12, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 12, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 12, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 12, 2016

TomK32 added a commit to TomK32/hledger that referenced this issue Jan 18, 2016

@TomK32

This comment has been minimized.

Copy link
Contributor Author

commented Jan 19, 2016

Okay, I did some work on my fork, sidebar with the accounts is a happy little sidebar now. What's not working yet is the transactions for one account. either we change /api/v1/accounts/:name to return a hash rather than an array or we rename it to /api/v1/accounts/:name/transaction. I'd be happy with either.

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Jan 24, 2016

I've merged your latest to hledger/master, with some cleanups - thanks.

Let's chat more about the api needs on #hledger.

@simonmichael simonmichael added the web label Feb 18, 2019

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 18, 2019

2019 update:

This issue led to the creation of hledger-api, a prototype JSON-only HTTP server, based on servant instead of yesod. It had a first release some months later on 2016/10/26, and I have been carrying it along since then. Maintenance has been non-negligible since servant brings in a new large set of dependencies. As far as I know, in the three years since TomK32's client prototype nobody else has got around to using it.

hledger-api could still be a very solid base to build a more extensive server on, but now I think maybe I should have just added JSON routes to hledger-web instead. And an option to turn off the server-side web UI. One package instead of two would have meant less work, easier installation, and also more exposure of the JSON api to folks installing for the server-side UI, and vice versa.

MoLe is a new mobile client, currently targetting hledger-web, which brings up a choice: whether to add missing API functions to hledger-api, or add them to hledger-web.

@real-dam

This comment has been minimized.

Copy link

commented Feb 19, 2019

Hi, MoLe author here,
It would be great if hledger-web could provide a stable JSON-based API. The current HTML-scraping works, but apart from the obvious fragility regarding future changes in HTML output, it can't guarantee that a posted transaction for addition is accepted, without parsing the most of the potentially huge response (POST /add redirects to /journal).

I imagine that a defined API would make this a piece of cake.

My preference is that the API is added to hledger-web, instead of hledger-api. In my use case (family accounting), I need hledger-web either way because some of the users/situations rather using the browser. I already have the infrastructure for hledger-web, it is even conveniently packaged for Debian.

I also imagine that synchronization between hledger-web and hledger-api wouldn't be a trivial task, while hledger-web has already solved this when multiple clients access it.

If I may be allowed for another wish, it would be great of the new API would allow for transaction modification and deletion. Too often it happens that an entered transaction is not quite right and a way of correcting it (other than entering a reverting transaction and starting over) would be very useful. Right now I just SSH to the machine and make the corrections using a text editor. This works, and to my surprise has no synchronization effects on hledger-web, at least as long as no one touches ledger at the same moment, but since the goal of MoLe is convenience, a way to do the correction via the API would fit nicely.

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 19, 2019

What are the minimum (and also, the nice-to-have) API operations needed by MoLE ?

Modifying existing transactions is a very old goal, and it's a bit tricky because of the text file format. But it's just a matter of programming.

simonmichael added a commit that referenced this issue Feb 19, 2019

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 19, 2019

I added a few to hledger-web: http://hledger.org/hledger-web.html#json-api

@real-dam

This comment has been minimized.

Copy link

commented Feb 20, 2019

Thanks, @simonmichael, and sorry for the delayed reply.

The minimum API needed is (1) something to list accounts (with balance), (2) transactions (with date, accounts and amounts) and (3) something for adding new transactions. That would cover the hledger-web (browser-powered) functionality.

Modify/delete would be a huge bonus.

Account list may be served by /accounts and transaction list by /transactions, so the only missing piece from the minimum is /add (or /new or whatever).

simonmichael added a commit that referenced this issue Feb 21, 2019

web: support adding new transactions via JSON PUT (#316)
A single transaction can be added by PUT to /add.
(I read that PUT, not POST, should be used to create;
perhaps the web add form should also use PUT ?)

As with the web form, the `add` capability is required (and enabled by
default).

Here's how to test with curl:

$ curl -s http://127.0.0.1:5000/add -X PUT -H 'Content-Type: application/json' --data-binary @in.json; echo

New readJsonFile/writeJsonFile helpers in Hledger.Web.Json
are handy for generating test data. Eg:

>>> writeJsonFile "in.json" (head $ jtxns samplejournal)
@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 21, 2019

@real-dam, cc'ing this latest commit message FYI:

A single transaction can be added by PUT to /add. (I read that PUT, not POST, should be used to create; perhaps the web add form should also use PUT ?)

As with the web form, the add capability is required (and enabled by default).

Here's how to test with curl:

$ curl -s http://127.0.0.1:5000/add -X PUT -H 'Content-Type: application/json' --data-binary @in.json; echo

New readJsonFile/writeJsonFile helpers in Hledger.Web.Json are handy for generating test data. Eg:

>>> writeJsonFile "in.json" (head $ jtxns samplejournal)

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 21, 2019

And here's that sample json for reference:

{
  "tcomment": "",
  "tpostings": [
    {
      "pbalanceassertion": null,
      "pstatus": "Unmarked",
      "pamount": [
        {
          "aprice": {
            "tag": "NoPrice"
          },
          "acommodity": "$",
          "aquantity": {
            "decimalPlaces": 2,
            "decimalMantissa": 100
          },
          "aismultiplier": false,
          "astyle": {
            "ascommodityside": "L",
            "asdigitgroups": null,
            "ascommodityspaced": false,
            "asprecision": 2,
            "asdecimalpoint": "."
          }
        }
      ],
      "ptransaction_": "1",
      "paccount": "assets:bank:checking",
      "pdate": null,
      "ptype": "RegularPosting",
      "pcomment": "",
      "pdate2": null,
      "ptags": [],
      "poriginal": null
    },
    {
      "pbalanceassertion": null,
      "pstatus": "Unmarked",
      "pamount": [
        {
          "aprice": {
            "tag": "NoPrice"
          },
          "acommodity": "$",
          "aquantity": {
            "decimalPlaces": 2,
            "decimalMantissa": -100
          },
          "aismultiplier": false,
          "astyle": {
            "ascommodityside": "L",
            "asdigitgroups": null,
            "ascommodityspaced": false,
            "asprecision": 2,
            "asdecimalpoint": "."
          }
        }
      ],
      "ptransaction_": "1",
      "paccount": "income:salary",
      "pdate": null,
      "ptype": "RegularPosting",
      "pcomment": "",
      "pdate2": null,
      "ptags": [],
      "poriginal": null
    }
  ],
  "ttags": [],
  "tsourcepos": {
    "tag": "JournalSourcePos",
    "contents": [
      "",
      [
        1,
        1
      ]
    ]
  },
  "tdate": "2008-01-01",
  "tcode": "",
  "tindex": 1,
  "tprecedingcomment": "",
  "tdate2": null,
  "tdescription": "income",
  "tstatus": "Unmarked"
}

This might will change in future. I suppose there should be some version number somewhere ? (Where?) The fields ending with underscore are dummy fields representing things that could not be serialised, which will be ignored when this json is parsed (details).

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 21, 2019

And Types.hs is where you can find out more about what these fields are.

@TomK32

This comment has been minimized.

Copy link
Contributor Author

commented Feb 21, 2019

The fieldnames in the json are definitely weird and would be easier to understand if you dropped that initial from the parent.

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Feb 21, 2019

That's an artifact of haskell's record system, and the use of automatically derived code which saves a lot of work. Sorry..

@simonmichael

This comment has been minimized.

Copy link
Owner

commented Mar 2, 2019

We now have two JSON api servers (hledger-api and hledger-web) and one client prototype/example. So I propose to mark this issue resolved. More specific issues can be discussed as needed.

@MisterY

This comment has been minimized.

Copy link

commented May 14, 2019

@TomK32, I assume this is more for B2B interaction rather than being human-readable.
I agree that JSON looks a bit too chatty and weird at the first glance but it contains all the basic data and is a sort of write-once ordeal to encapsulate in some module and forget about it (until 500 errors tells you that spec has eventually changed on the server-side, in the future).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.