Understand parsing JSON

[voddan]

I cannot understand this part in http://play.witheve.com/#/examples/http.eve

search @http
  [#request #test-data response: [json]]
  json = [#array]
  lookup[record: json, attribute, value: [id name email phone website address: [street suite city zipcode]]]

commit
  [#user id name email phone website address: [street suite city zip: zipcode]]

What happens in lines 2, 3, 4? I was unable to find any documentation regarding lookup. Also, what structure does a http response have? What is the meaning of [#array]?

Thank you for any help

[Cormac-Williams]

Hi Voddan,
I was leaving this to be answered by the team, but then realised it was Thanksgiving in America and they were unlikely to be about much :-)

I'll take a crack at some of it. If I use slightly incorrect terminology some-one please correct me,

One of the abilities of Eve that tripped me (And I assume other people) up is the automatic assignment to unbound variables in a query if they share a name. You see this using json in the second line and a lot in the lookup. Once you get used to this part of the language though, it is a HUGE saver from writing boilerplate code.

Response is a record, attached to the response attribute of the request record. It can have two attributes in this case, body or json. The http database is detecting that the content type is json and is helpfully running the body text though a json to eav function that turns the json into Eve data structures.

The body text is still available and could be accessed using
[#request #test-data response: [body]]
In which case it will be a plain old string.

The second line is basically saying, "Query the database for records tagged both request and test-data that have a response which has a json section. Assign this json section to the variable json.

The third line ( json = [#array] ) is an equivalence, it is a filter that says "The json record must have an #array tag". Since json can have both object and array formulation, it is a guard that the response has been turned into an array list.

And finally the Lookup, you correct in that it hasn't been fully documented yet, and I'm not sure that I fully understand all its abilities (It's quite powerful). But essentially this is another blended query and assignment, it is saying "For each record in json, lookup the attributes and assign their values to the following unbound variables."

I hope that mind-dump helped and if you have any further questions do not hesitate to ask either here or on the Google group.

Regards,
Cormac.

[voddan]

Thank you, @Cormac-Williams, that was very informative!

Let me repeat what you said to see if I've got it right:

• Line 2 and 3 are searching for requests that has a json member in their response. That means lines 2, 3 may be replaced with a [#request #test-data response: [json: [#array]]] (it actually works)

• Line 3 is there because this particular JSON response is an array of structs, so EVE parses it into a struct with #array tag.

• If there were a way to turn this array into a temporary database, there would be no need for lookup. The code would be:

  search @http
    [#request #test-data response: [json: [#array]]]
    json >> @json  // Imaginary synax
  
  search @json
    [id name email phone website address: [street suite city zipcode]]
  
  commit
    [#user id name email phone website address: [street suite city zip: zipcode]]
  

Are those correct?

[voddan]

And since we are at it, where can I find a description of standard databases (@http, @browser, @view)? I would need it to know about response: [body]

[Cormac-Williams]

Hi Voddan,

I think you've got it, except that the json in some ways is better thought of as the result of a query. It has already been parsed into an eve record structure for you. And can be further queried by the search section. i.e.
adding
name = "Ervin Howell"

to the search section works as a filter, even if you add it prior to the lookup line.

I read the code to try and figure out whatever has not been documented and assume that it can change under me. There's also very useful discussion and links to examples that goes on in the https://groups.google.com/forum/#!forum/eve-talk group.

The code for databases is in the src/runtime/databases and some further documentation on writing functions and databases is here.
https://gist.github.com/joshuafcole/52f706e0e838b3fb7536814de8879af0

The @view functionality is written in Eve and is in the examples/view.eve file.

Regards,
Cormac.

[voddan]

Wow, cool! I've seen view.eve, but I haven't realised what it was. IMHO this comment should be at the top of view.eve.

By the way, @view uses tha [value | foo] syntax. What's this | ?

[Cormac-Williams]

An undocumented language feature that the team is not sure will stay ;-)

I've been thinking of it as a shorthand that says "At this point I've told you enough to identify a single unique record, use that.". But I could be completely wrong on that.

Regards,
Cormac.

[voddan]

Is there a way to work with @view without | ? Could you give me an example? What I tried did not work.

[Cormac-Williams]

Should be, post your example and what you are trying to do and I'll take a look.

[voddan]

For example I've tried to use @view for displaying debug values:

commit
  [#api-url url: "https://api.github.com/repos/jquery/jquery/stats/contributors/"]

search
  [#api-url url]

commit @view
  [#value url]

This code has no effect thou

[Cormac-Williams]

Try
commit @view
[#value value:url]

And see what happens ;-)

[voddan]

It works! Thanks!

Then I don't understand why EVE did not stop me from committing a record with an untagged value. This must have been a error, or it must have worked.

[Cormac-Williams]

It did work, url was bound during the search phase and then committed to the @view database as part of the record. Try doing the same thing with an unbound attribute and you will get errors.

[voddan]

No, it does not give me an error. The whole problem was that my initial code with commit @view [#value url] produced no errors, nor results.

[Cormac-Williams]

Your initial code was the below?

First block. This commits a single record tagged api-url and one attribute url with its corresponding value.

commit
  [#api-url url: "https://api.github.com/repos/jquery/jquery/stats/contributors/"]

Second block,

search
  [#api-url url]
commit @view
  [#value url]

In the search section, this searches for records tagged api-url and binds the url attributeof the record to the variable url.
In the commit section, you commit a record tagged #value and the previously bound url variable. In the commit section, if you tried to do this

commit
[#value url-this-wont-work]

You should get an error saying "Nothing is providing a value for url-this-wont-work" pretty much immediately.

If this isn't how things are working and you are not getting this error, then post your full Eve file somewhere and I will try to recreate.

Regards,
Cormac.

[voddan]

I think I did not explain the situation clear enough. Yes, my code is what you wrote. This is my whole file, there was nothing besides those 2 blocks.

When I said that it did not work I meant that there was no visible result. I would expect that [#value url] and [#value value: url] have the same result since the "untagged" version [#value url] has the correct structure.

But this is not true. My guess is that the first has a "type" like [tag, unnamed], while the second is [tag, value]. So the first does not work as expected , while the second does work.

So my question is: why there is no error when committing a record with a member without a name? Is there a situation when it can be used?

[Cormac-Williams]

Ah, I understand. There is no formal type system underneath Eve at this time, everything is EAV (Entity, attribute, value) under the hood. This includes tags #value and #whatever, they get turned into a "tag" attribute with the values of the tag names in the datastore.

So, if you do something like.

commit @view
  [#value url: "123"]

That is perfectly valid Eve code. The reason it didn't appear to begin with is that the view.eve file searches for records in the view database that are tagged #value and have an attribute value. Since your original commit to the @view database did not have a value, it didn't come up in the search.

As to whether there is a situation it can be used? Well yes. The freedom to copy and augment only the data you are interested in from one place to another is to an extent the essence of programming and computers :-) Eve is just going about it in a much more "Database" and "Query" oriented fashion than most languages you come across.

Regards,
Cormac.

[voddan]

Sorry, I guess I din't get the last part.

Does that meant that [#value value: url] is equivalent to [#value url: url]? That makes no sense to me. Why would one want to use such a syntax sugar?

[Cormac-Williams]

No, they are different. Both are records that have a tag #value, one has an attribute value, one has an attribute url.

[voddan]

Sorry, I meant [#value url] and [#value url: url]. What about them?

[Cormac-Williams]

If you have url previously bound to value(s) then they are the same. the [#value url] version is simply syntactic sugar for the latter.

[voddan]

Ok I see. This is extremely uncomfortable not to be able to freely choose names for my "local variables". 😞

[Cormac-Williams]

You can, the syntactic sugar is for ease of use purposes, there is no requirement to do so. But how many times have you typed the same name on both sides of an assignment just to move one thing to another?

[voddan]

@Cormac-Williams Thanks for explaining all this!
I hope some of it will eventually go into EVE docs.

[cmontella]

@Cormac-Williams Thanks for filling in over the holiday! :P Your explanation was pretty much on point, so nothing really to add there. I would like to address a couple things though:

1. This is one of the more esoteric corners of the language right now. The JSON/HTTP example was more of a proof of concept than a "best practice" of working with JSON in Eve, but right now it's the only practice so it will need some documenting. I've opened some issues to track these:

• Parsing JSON - Document JSON parsing docs#32
• lookup[] - Document lookup[] function docs#30

2. Regarding the pipe |, to see why it's needed, consider a player in a game. It can have a position on the game board, specified by an x and y location.

[#player x y]

According to base Eve semantics, the record [#player x: 1, y: 1] is different from [#player x: 1, y: 2], even though we can conceptually know that the player is the same even though it's at a different position in the game. The pipe tells Eve that every attribute to the left of the pipe contributes to its identity, while everything to the right does not. So if we write the player this way

[#player | x y]

Then Eve knows that the location of a #player doesn't contribute to that player's identity.

Is there a way to work with @view without | ? Could you give me an example? What I tried did not work.

The pipe here is used in a similar way. We say [#value | value] because in a view, we'd like to track how the value changes over time. This has the same problem as #player, because we want the view to be the same view even if its value changes. Without the pipe, the view blinks as value changes, because the view is removed and replaced instead of being changed in place.

3. Ok I see. This is extremely uncomfortable not to be able to freely choose names for my "local variables".

Just wanted to make sure there was no confusion here. If you are happy with the names the record uses, then you just pull it in:

search
  [#foo name]

Then you have the ability to use name as a variable. If you don't want to use name as a variable, you can reference it directly

search
  foo = [#foo]
  [#Corey name]

bind @browser
  [#div text: "My name is {{name}}, and Foo's name is {{foo.name}}"]

And finally, you can re-name any attribute on a record to use as a local variable

search
  foo = [#foo name: foo's-name]
  [#Corey name: corey's-name]

bind @browser
  [#div text: "My name is {{corey's-name}}, and Foo's name is {{foo's-name}}"]

So you have complete control over how you want to use names. Let us know if you have any more questions @voddan

[cmontella]

I'm going to close this, since I think your issue has been solved.

[Cormac-Williams]

Glad to help :-P

[voddan]

@cmontella Thank you for the clarifications!

I feel that at the moment EVE has a bunch of features that are rarely used (< 10% of apps). That may significantly hurt the learning curve. Maybe there are ways to generalise them somehow? For example EVE has very powerful tools for searchers, but in few cases you use lookup, which is a totally different concept. IMHO lookup must be eliminated or somehow generalised.

Also could you please update https://witheve.github.io/assets/docs/SyntaxReference.pdf with all new operators / syntax constructs? This document at least tells me how different features are called, so I can try googling them. Right now nor |, neither lookup is presented there.

[voddan]

@cmontella I have one more question if you don't mind.

You pointed out that in my example the view blinks if used with search .. bind [#value value]. I wonder why is that that bind has to replace the record (delete than recreate) instead of updating it. How is the identity of a record is important?

Thanks

[empath]

Let's say I have some arbitrary json, can someone walk me through exploring it interactively with eve? Let's say I don't know anything about what the structure of the response will be ahead of time?