Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Hypermedia #13

Open
zdne opened this Issue · 115 comments
@zdne
Owner

Direct support for hypermedia and modeling actions & relations.

@rurounijones

I am surprised that this is marked for 1B, imho there are other, more real-world use issues that would be better implemented in 1B and leave Hypermedia until 1C

@zdne
Owner

I believe Hypermedia elegantly solves many real-world issues such as client decoupling, versioning (along with content negotiation), discoverability to name a few. It also directly contributes to better API design, maintainability and scalability.

The idea is to bake its support directly into API Blueprint so you do not even have to think whether you are hypermedia or not. For example this could automatically provide the location HTTP header where applicable or construct the link HTTP header etc. In cooperation with #10 this could go even further utilizing media types such as Collection+JSON, Siren or HAL.

Idealistically, should you choose to, I would like you to think about resources, relations and actions not HTTP verbs, URIs and such, but that is definitely even beyond the 1C..

@mikekelly

Has anyone sketched out what this might look like?

@zdne
Owner

@mikekelly, this is how I project it at this very moment:

In API Blueprint (Markdown), using some semantic assumptions:

  1. Define a resource with a name, id or an URI
  2. Describe the resource model media type (state, attributes)
  3. Describe relations between the model and other resource models
  4. Specify a transaction example with a hypermedia affordance provider media type content negotiation

The API Blueprint parser (or its harness) will take the care of injecting the actual model media type and relations (possibly even actions) into the hypermedia affordance provider media type.

Using part of @lukestokes foxycart API something along these lines (again, just sketching here):

# Resource User [User]

+ Model (application/json)

    ```
    {
      "first_name": "Qing",
      "last_name": "woldu",
      "email": "test481598556@example.com",
      "phone": "12345678",
      "affiliate_id": 0,
      "is_programmer": false,
      "is_front_end_developer": false,
      "is_designer": false,
      "is_merchant": false,
      "date_created": "2012-02-29T13:55:09-0800",
      "date_modified": "2013-07-11T15:28:56-0700"
    }
    ```

+ Relations
    + self = [User][] ... This User
    + stores = [Stores][] ... Stores for This User

## Retrieve a User [GET]

+ Request (Accept: application/json)
+ Response 200 (application/json)

    [User.json][]

+ Request (Accept: application/hal+json)
+ Response 200 (application/hal+json)

    [User.hal+json][]


# Resource Stores [Stores]

+ Model (application/json)

    ...

The API Blueprint parser would take care of injecting the user application/json model into the respective response media type in its resulting AST (and AST media types) e.g. do just a verbatim copy for [User.json][] or build the application/hal+json representation in the case of [User.hal+json][] using the model and relations.

Note I would also love to have Siren on board alongside with HAL in the first round. Considering other provides later. Also this concept can be extended to auto generate the Link HTTP header.

This is in a nutshell how I think about this issue at the moment. There are still few pieces of puzzle missing but that is alright for now. Please let me know what you do think and whether it can work at all.

Thank you!


Note for @kevinswiber : This is the topic we didn't have the chance to discuss at #apicraft.

@lukestokes

I'll let smarter people than myself comment on the approach, but let me just say I'm really excited about seeing progress in this direction! Our sandbox supports both HAL and Siren, so I hope it can be a useful "real world" test bed for trying some of this stuff out. If I can help in a specific way, please let me know. Here are some of the keys I can think of as it relates to the needs client developer:

  1. Educate and enforce the importance of REST principles (think in terms of media types, response codes, caching concerns, resources, link relationships, etc and less in terms of remote procedure calls to URLs).

  2. Use visualizations in the documentation that help them navigate relationships to find the resources of the API they are interested in. As a real world example, we have users who only want to integrate with the API so they can generate and manipulate coupons. Building a documentation interface that helps them navigate to that relationship will be great.

One thing I was also considering is that not every API has to (or will be) a true REST API. If there's a clear way to distinguish between JSON RPC APIs and REST Hypermedia APIs, that would be fantastic.

Great work, Z. I'm really looking forward to where this goes.

@zdne
Owner

@lukestokes I do agree in all points but the very first one – you are very qualified, if not the best, for commenting on this issue!

Regarding the clients – as we have discussed with @cainus its the clients where we need to improve the situation next. Not by telling them what they should do but pointing real life benefits as well as offer the best tools to consume our APIs. Now imaging sticking an hypermedia-enabled API Blueprint into a client SDK to configure it for your API and provide realtime request checks etc... (OK now I am totally going both off topic & off the roof :)

@mikekelly

I think the only reasonable way of achieving broader adoption is to ship APIs where clients will code against them without even knowing what hypermedia is. This means we need media types that people can treat as plain JSON and they are still just as easy to work with and understand.

If I was you, I would pick the media types you initially adopt here carefully. If you offer something that is too complicated, and not intuitive enough - you are going to undermine your larger objective.

But I would say that, obviously. :trollface:

@zdne
Owner

Mike, while I do agree with this line:

ship APIs where clients will code against them without even knowing what hypermedia is

However I do believe a broader adoption would be achieved through providing API SDKs and/or tighter framework client integration rather than finding the "holy grail" of media types. I treat a media type just as a mean to transfer the resource representation and, in some cases, hyperlink relations. Idealistically a client should be interested in (and exposed only to) actual resource data and the relations, not in the technical means of the communication itself.

In another words as a client you should only need to know your data are and where to get the next.

This is exactly what I would like to achieve with API Blueprint. In the first step to sweep the technical means of a hypermedia API under a nice and tidy Markdown carpet, so you do not have to really think HAL, Siren or anything like this – you just focus on your data and relations. Second do the same on the consumer's end – using an API Blueprint on client to hide the HTTP / transfer technicalities and provide directly the data and relations.

What do you think?

@lukestokes

It's funny, prior to the RESTfest talk by @bsletten , I would have been worried about hiding things behind an SDK or client library. But now, if REST is truly used under the hood, I can see that the format and the datastore don't matter as much, as long as information can be treated as addressable resources which tell you what can be done next. His keynote video isn't up on vimeo yet, or I'd link to it.

@kevinswiber

@zdne

However I do believe a broader adoption would be achieved through providing API SDKs and/or tighter framework client integration rather than finding the "holy grail" of media types. I treat a media type just as a mean to transfer the resource representation and, in some cases, hyperlink relations. Idealistically a client should be interested in (and exposed only to) actual resource data and the relations, not in the technical means of the communication itself.

:+1:

We care more about the media types, because that's what we do. Most people just want to get the job done.

The difficult thing when supporting multiple hypermedia types on the same server is transforming capabilities between media types. If one media type only supports links while another supports controls, do you whittle down to the least common denominator? If so, why have additional media types? Do you provide enough information for capabilities of ALL media types supported? If so, do you just discard that information when rendering media types that don't support it?

These aren't very easy problems to solve in some cases. I'm curious what route @lukestokes took with FoxyCart.

@kevinswiber

Note for @kevinswiber : This is the topic we didn't have the chance to discuss at #apicraft.

Hey @zdne, sorry we didn't get a chance to discuss.

Here's how I've been looking at Siren APIs, in terms of documentation.

Things to define:

  • Classes
  • Link relations (for entities and links)
  • Actions

Entities of a particular class may have certain properties, sub-entities, actions, and links. (Note: An entity can have multiple classes.) Defining this is important in documentation. I use class values to route responses to appropriate handlers on the client side. This allows the client to respond to the actual message from the server instead of setting an expectation at request time. For some, this is a different way of thinking, but it's an extremely powerful, reactive pattern.

Everything flows from the entity class documentation. Sub-entities have link relations that should be defined. Actions should be defined, and so should expected input fields. Hidden input fields, of course, should not be defined. The Siren spec instructs clients to resubmit hidden fields.

As far as functional documentation (docs that are able to submit requests), with Siren, this has to be dynamic. One of Siren's benefits is communicating available actions at request-time, based on resource state. Potential state transitions can change, and so, too, should the active docs.

@lukestokes

These aren't very easy problems to solve in some cases. I'm curious what route @lukestokes took with FoxyCart.

We are just at the lowest common denominator right now. I wanted to include actions in our Siren support (and still plan to in the future), but we didn't want to go there until we finalized more important link relationships and property naming conventions.

I also want to support Collection, though I'd probably have to do an extension of it because we want to include properties of the collection such as total_items, returned_items, limit, and offset. In the future I may look into XHTML as well.

The formatting of the output, imo, isn't nearly as hard as properly designing the API in the first place. As long as the API has a good format layer in place, it's just a matter of moving data around. We did run into some funkiness with how things are done in JSON vs. XML, but for the most part, I think we came up with good approaches that wouldn't appear as compromises. Our hope in supporting multiple formats is that we can be a "real world" API for people to start testing their hypermedia tools against. More tools = more adoption.

Feel free to play around with it. I'd love any feedback you have: https://api.foxycart.com/docs

Any way, as it relates to this discussion, maybe it would make sense to include @mamund H-Factor ideas when thinking about the documentation system. Just a thought.

@dilipkrish

I'm just dipping my toes into this hypermedia ocean. The featured apiary apis look great, hadn't discovered it until recently! Apiary looks really powerful! @lukestokes the foxy cart api looks like it already supports hypermedia and consequently serendipitous api navigation discovery!

Anywho, as a background, I've been trying to socialize RESTful services and take it to the next level (of maturity) @ The Container Store. As a start, we've been using swagger to "document" our internal API's. Thats a great start, and it seems like a very similar approach to apiary. I've been thinking about how to make the simple API's hypermedia aware, and secondly how to make them discoverable once it is. Like @mikekelly said,

I think the only reasonable way of achieving broader adoption is to ship APIs where clients will code against them without even knowing what hypermedia is.

My 2 cents; considerations when it comes to hyper media, documentation and restful services

  • The generation of the documentation cannot be deliberate, as in, distinct from the act of designing the API/service itself. So either the service can allow the generation of service information or vice versa. That way each side does not have the risk of being out of date. So the workflow is really important for adoption.
  • Secondly, the whole notion of "documentation" falls flat with respect to a hypermedia API. By definition for a hypermedia API you should only need to document the entry points (as proposed by @zdne) the rest should be inferred as a result of navigation and discovery. Otherwise, IMHO it smells of the RPC style documentation that we got with SOAP, WSDL and its band of brothers.

Glad to help take this discussion forward.

@mikekelly

@kevinswiber

We care more about the media types, because that's what we do. Most people just want to get the job done.

I'm in the latter camp for sure. Hal happened because I wrote a gist to show what I was building, then bunch of other people got interested so made a little website, then people started asking for a spec...

I don't care that much about media types - I care about helping people build effective, usable APIs.

There are some media types which I, as someone who's been around this space for a bit, feel are basically over-complicated and solving problems nobody's even sure they have - which is fine in and of itself, but when it comes to projects like this is worth highlighting as a risk for the reason you've outlined here:

The difficult thing when supporting multiple hypermedia types on the same server is transforming capabilities between media types. If one media type only supports links while another supports controls, do you whittle down to the least common denominator? If so, why have additional media types? Do you provide enough information for capabilities of ALL media types supported? If so, do you just discard that information when rendering media types that don't support it?

Don't get me wrong, I think pushing things further is worthwhile, but I think picking the low-hanging fruit first and then figuring out where to go next is a more reasonable approach. It sounds like this is exactly what @lukestokes has done with the foxycart API - I'll be interested to see where they go next.

@zdne
Owner

@mikekelly

I care about helping people build effective, usable APIs.

Excellent point Mike! This is my aspiration with API Blueprint.

Again looking at FoxyCart HAL Browser: Basically my idea is to describe what is in the left pane in a Markdown syntax and let API Blueprint parser generate what is on the right. Without any prior knowledge of HAL or HTTP / REST.

Just tell me about your data and the relations. Simple.

@zdne
Owner

@kevinswiber

Thanks for your feedback and tips on Siren, much appreciated!

Indeed if we decide to go for both the approach of @lukestokes with the smallest common denominator is the way to go.

Now, it seems to me the (choice of) (hyper)media type is really more important when you have one end open. Should there ever be something like an API Blueprint driven client consuming an API designed with API Blueprint then the whole (hyper)media type would be really just a 'technicality' under the hood.

@mikekelly

Yes that sounds like a really good idea. fwiw, when I spiked the haltalk demo - this is what the link relation documentation ended up looking like:

http://haltalk.herokuapp.com/rels/signup

@zdne
Owner

Yay! This looks almost like an API Blueprint:


signup

Create an account [POST]

  • Request

    • Headers

      Content-Type: application/json
      
    • Body

      {
        "username": "fred",
        "password": "pwnme",
        "real_name": "Fred Wilson"
      }
      
  • Response 201

    • Headers

      Location: -----
      

Very helpful. Thanks for sharing!

@kevinswiber

@mikekelly

I'm in the latter camp for sure. Hal happened because I wrote a gist to show what I was building, then bunch of other people got interested so made a little website, then people started asking for a spec...

You're in both camps. Everyone has a stake in their work, and hopefully, the belief that it's helpful.

There are some media types which I, as someone who's been around this space for a bit, feel are basically over-complicated and solving problems nobody's even sure they have

Siren isn't an academic project. It came out of real work, as well. It was solving my problems. Now I'm helping others use it to solve their problems. Siren does have capabilities HAL doesn't. It's okay if you don't need them. Other people do. This conversation between us is a broken record that is being played on 3 different mailing lists. I'd like to show @zdne some respect and not pollute this thread with nerd battle. I'm happy to take this to the Siren list if you'd like to play this song together once more.

@kevinswiber

http://haltalk.herokuapp.com/rels/signup

@mikekelly @zdne

So a question I run into from time to time asks if something like this is actually over-specified. The usage of HTTP headers and status codes is already well-defined. You could really just say "Follow HTTP" for a lot of this.

Also, there are some things missing that I'd care about as a client developer. What about validation? When might there be an error condition? Are usernames unique?

I'm also starting to favor leading with description rather than HTTP, but that's mostly a formatting thing. I'd prefer to highlight "Creates an account" over "POST," because that's the bit client developers are really going to care about.

@mikekelly

I'd like to show @zdne some respect and not pollute this thread with nerd battle

great, me too! It's not a personal affront, I'm just offering advice that I think makes sense for this project at this point in time.

Also, there are some things missing that I'd care about as a client developer. What about validation? When might there be an error condition? Are usernames unique?

Yep - there are many more things this description document could contain, haltalk is a really basic demo - it's not a finished product.

The main point of the demo is to show that the various transitions of an API can be broken down into, documented, afforded, and discovered via link relations. It's a no-nonsense strategy for making an API discoverable and moving the attention away from URL patterns.

@kevinswiber

Now, it seems to me the (choice of) (hyper)media type is really more important when you have one end open. Should there ever be something like an API Blueprint driven client consuming an API designed with API Blueprint then the whole (hyper)media type would be really just a 'technicality' under the hood.

I want to say yes, because this sounds interesting, and anything that sounds interesting is worth doing, in my opinion. Innovation is a big, messy pile of ingredients. Sometimes you just need to throw 'em in the cauldron and see what comes out.

Media types offer different capabilities, and that's where it gets tricky. No one cares too much about links vs _links. When you're looking at consuming an API, what you care about is features offered by the provider. I think we all agree on this.

The question then becomes, as an API provider and potentially someone who will create clients for various platforms, "Which media types help support the features I'm looking to implement?" This is where making a mistake really kind of stinks. You can start off with something minimal and try to bolt-on later, but eventually, the Frankenstein monster emerges... or you intentionally cripple the capabilities of your API. (Those words sound harsh, but it's not always so terrible.)

So the argument for starting with a media type that has the most capabilities is there. Hydra[1] is another media type I've taken a liking to lately. I need to examine it more closely, but I would submit Hydra as a candidate for another one of these "capabilities-rich" media types.

Of course... you could always come up with whatever format you think would work for you in this situation and birth something like application/blueprint+json. :)

You should also take a look at ALPS[2] if you haven't already. It's aiming to add supporting application semantics to APIs.

I feel like CCing some people since I'm talking about their work, and surely, they know more about their work than I do. /cc @lanthaler @mamund

Now... light that fire and get that cauldron boiling. ;)

[1] http://www.markus-lanthaler.com/hydra/spec/latest/core/
[2] http://alps.io

@lukestokes

When I started thinking about docs, I figured people building their businesses on our platform will want more information rather than less. I've only got some twig templates in a twitter bootstrap theme, but I thought about having a short paragraph summary of the link relationship along with the affordances (we're a CRUD-heavy api so one relationship does multiple things via various HTTP methods, something OPTIONS can also help discover). Additionally, I want to show the property with detailed descriptions, types and constraints. We also want to show some example outputs in the various formats we support. It's a work in progress, but here are some examples:

http://api.foxycart.com/rels/stores
http://api.foxycart.com/rels/transaction
http://api.foxycart.com/rels/create_client

When it comes to embedded resources, I also include stuff from /rels/store inside of /rels/stores. Something to think about as users may work with a collection in a slightly different manner than a single resource.

I don't think there will ever be a "one-size-fits-all" solution to everyone's needs, but at the very least, focusing on link relationships instead of URLs will be a big step forward, IMO. As long as things are extensible so I could add in my own details, that would be great.

Really excited about where this is going, guys. Thanks for encouraging me with your efforts. :)

@kevinswiber

http://api.foxycart.com/rels/stores
http://api.foxycart.com/rels/transaction
http://api.foxycart.com/rels/create_client

Hey, I think this looks pretty good!

Capturing all the elements you have outlined there in a more abstract way would get something closer to the hypermedia version of WADL.

@lukestokes Thanks for actually implementing a real hypermedia API with public documentation. It's helpful for the whole community and hopefully helpful for FoxyCart, as well.

@lanthaler
@mamund

I know this thread is getting quite long; good stuff here. I wanted to add one more thing to the mix when considering how to go about documenting a "Hypermedia API"

I've recently gotten into the habit of documenting the actions, not the objects. I have a (very) rough example posted in another repo[1]. Right now this approach takes a couple steps (not ideal):

  • doc the media type
  • doc the domain
  • provide examples (combine the media type and the domain)

I think that is too unwieldy and can be greatly improved. However, my point here is to pitch documenting the hypermedia controls, not the resources. Its a very diff approach, but something I think can work well. It allows servers to decide their own URIs, their own representation details (what data and which controls appear in responses) and still gives client devs enough to write a working client (that recognizes both the data and the controls when they show up in a response).

Anyway, its just another approach to consider.

Cheers.

[1] https://github.com/apiacademy/class-scheduling/blob/master/docs/problem-domain.asciidoc#actions

@mikekelly

Which media types help support the features I'm looking to implement?

More features in the media type does not translate to more features of a service. Compromising on media type "features" and instead, folding that complexity into something else (e.g. link rel documentation) does not make a service "inferior". It is a design decision.

more can be less.

@mikekelly

Defending complexity by offering more "features" is a well proven sales technique, though - I'll grant you that.

@kevinswiber

@mikekelly I acknowledge your opinions. I have a feeling individual stubbornness on both our parts will prevent compromise. You admit HAL is intentionally minimalist yet claim anything more complex is a waste. For some reason, you think HTML is the only other media type that gets to play in the game. You haven't recognized any positive aspects regarding any other hypermedia type. That is not very constructive to the conversation. I get it. You want everyone using HAL exclusively. That won't happen, in my opinion, because of its minimalist design. I can program with GOTO all day long, but you know what? Sometimes I like IF-ELSE structures, too. You are a very loud minority, and your opinions, though acknowledged, are losing value.

@kevinswiber

@mamund This approach is very interesting. Are the actions for a link ID always static or could they change at runtime?

@mikekelly

You admit HAL is intentionally minimalist yet claim anything more complex is a waste.

I would be less inclined to claim that if the complexity in question is presented along with a demonstration of the additional (evolvability-related) behaviours it affords, rather than hand-waving and talking about "having more features".

You are a very loud minority, and your opinions, though acknowledged, are losing value.

This strikes me as a bit unnecessary, but I'm sorry you feel that way.

@mamund
@zdne
Owner

@mikekelly @kevinswiber

Gentlemen,

I am really happy to have both of you in this discussion but please allow me to turn your precious attention to the question that is slowly building in me ever since I have started thinking about the Hypermedia support in API Blueprint:

Do you think that another level of abstraction above the REST (and Hypermedia) can actually work out? Didn't Hypermedia failed (in the terms of adoption) just because its already too abstract for many?

I feel like both @mamund's ALPS and ours API Blueprint are (by definition) abstract from HTTP and REST and now we basically want to sweep the Hypermedia under the carpet. Which from one point of view be looked at as a simplifying of things; yet on the other side it means increased complexity through abstraction?

In another words, can we abstract from REST enough to provide a simpler way to define, build and consume our services (that would be under the hood Hypermedia, HTTP and TCP/IP driven)? Should we even try?

@kevinswiber

@zdne

Yes, and yes!

For most developers, utilizing remote services is not about REST or hypermedia. The protocols, media types, and architectural stylings are all means to an end. If the focus is on providing a good experience for developers to provide and consume remote services, I think this is very doable. The underlying benefits of REST should remain, but again, these are details under the hood.

Not to scare anyone away, but this essentially mimics the "Generate WSDL"/"Import WSDL" ease of use on top of protocols that are infinitely more scalable and hopefully with "generate"/"import" features that don't require extreme maintenance to avoid breakage (evolvability, decoupled implementation stuff).

@mamund
@lukestokes

@mamund

Make the documentation centered around the hypermedia controls; not the resources.

Very interesting (and thank you for linking to an example). Question about your example... does it assume a flat structure? That, to me, is the missing piece... if, as a client developer, I want to do something, I have to understand how the relationships between resources work in order to get to the right place, right? We're back to WWBD (What Would Browser Do) and thinking about the UI/UX aspect of how the API will be used. Or maybe the client can figure this out on their own as long as it understands the domain and the controls?

I do think any step away from just thinking in URLs (which, I guess would be thinking purely in resources) and step toward thinking in terms of controls / actions / link relationships would be really helpful.

@zdne
Owner

@mamund

Pardon, I have expressed myself wrong – I want to sweep the Hypermedia technicalities (headers, media-types etc.) under the carpet so we can actually focus on and leverage the hypermedia qualities.

Indeed I want to promote the Hypermedia way of thinking; just maybe without explicitly saying it. I still need to think about your controls-first approach. It is intriguing indeed. It suddenly feels odd living in the model & relations world.

As you are definitely ahead by far in this field I will try to enjoy the luxury and learn from your example.

@zdne
Owner

@kevinswiber

Thanks Kevin! Much appreciated. I am not biased with WSDL so that definitely won't scare me away! (Albeit I strive to learn from the old mistakes)

@mamund
@mamund
@mikekelly

I feel like both @mamund's ALPS and ours API Blueprint are (by definition) abstract from HTTP and REST

Hm.. but , iirc, the blueprint response description examples are actually expressed in terms of HTTP?

now we basically want to sweep the Hypermedia under the carpet. Which from one point of view be looked at as a simplifying of things; yet on the other side it means increased complexity through abstraction?

I think you should focus on building stuff that helps people make sense of the actual messages that are going over the wire, because this is the stuff that people will poke around with, discover, debug, etc. I haven't seen many people asking for more abstraction in APIs recently, but I have seen them asking for more accurate documentation and less breakage of their client code.

@zzen
Owner

For me +1 for @mikekelly. For me the "lessons learned" from SOAP/WSDL is "don't use abstraction as an excuse to make things complicated under the hood". Because at the end of the day, when sh*t hits the fan, under the hood is where you'll end up, poking around with some sort of debugger.

But things could work a bit more more out-of-the-box. Think (optional) Rails-style convention-over-configuration.

@kevinswiber
@mikekelly

Tucking it behind unspecified formats for link relation documentation does not make it disappear. You hit the problem later than sooner, but you still hit it nonetheless

What do you mean by "unspecified formats"? Do you mean an HTML page of documentation written in plain English? HTML is already specified and most people are fine reading English (or some other language). Having said that it would probably be useful to standardise on a template for how link relation documentation should look, just so people have less to think about. I think that is something this project could establish nicely and it sounds like @zdne is already working towards it, anyway.

Nobody said link rel documentation will make the complexity of a service disappear, but it does draw a very clear line in the sand where the parts of the API that are intended for machines belong (the API message) and where the parts of the API intended for developers belong (the link relation documentation). Adding more controls into the message just muddies where that line is drawn, and it doesn't really buy you anything wrt machines.

If you need forms because your API is actually intended as a GUI, then serve your controls as HTML (e.g. disqus comments). HTML is everywhere and it's very well understood - there's really no good reason to try and reinvent its functionality in JSON.

@kevinswiber

@mikekelly Again... automation in dynamic workflows is not possible without an affordance that communicates available state transitions at response time. With Siren, I'm evaluating moving away from HTML-defined types for input fields. There are pros, but there are also cons. GUI benefits just come free. The benefit during M2M is runtime validation. This means a field type can change on the fly from 'text' to 'email' and automatically reap data cleanliness benefits. A client can deny input before it even submits the request. There are other ways to do this, and that's what I'm planning to evaluate.

Putting this information in the message has the added benefit of ease in developer exploration. Just surfing the API, a developer can learn what is needed to continue down the path toward a goal... just like machines, but able to make smarter decisions or change directions once in motion.

By "unspecified formats," I meant there's typically no link relation documentation that is automatically consumed by the client for the purpose of dynamic runtime client configuration. This is what would be required for the workflow use case that doesn't include this information inline. This sort of thing could certainly exist some day. Without something like this, I haven't seen a good design for this use case that doesn't involve inline actions of sorts.

These are the benefits I need, and it's easier for both machines and developers if the information is included in the message body itself (no extra requests required).

I often code in state machines, and I like to avoid dummy requests. If the server communicates available transitions, along with metadata describing the parameters for executing those transitions, this reduces network chatter and makes "where to go from here" crystal clear. That's beneficial for a number of reasons.

@mikekelly

automation in dynamic workflows is not possible without an affordance that communicates available state transitions at response time.

Couldn't you use links for each of the transitions? Why would that not be sufficient?

With Siren, I'm evaluating moving away from HTML-defined types for input fields. There are pros, but there are also cons. GUI benefits just come free. The benefit during M2M is runtime validation. This means a field type can change on the fly from 'text' to 'email' and automatically reap data cleanliness benefits. A client can deny input before it even submits the request.

Well, in that case you end up with less visibility over which clients you just broke. You could just handle this server side and respond with a validation error message. The basic behaviour is equivalent (the client will break) but approach is better for the client because:

  1. Their lives are easier because they don't need to implement pre-validation.

  2. You have more visibility into the effect of your change, since you can log invalid requests that failed server side - and track how quickly those clients get fixed.

This is a really good example of where adding complexity and "features"  actually adds no value and arguably does the opposite.

Putting this information in the message has the added benefit of ease in developer exploration. Just surfing the API, a developer can learn what is needed to continue down the path toward a goal... just like machines, but able to make smarter decisions or change directions once in motion.

And yet link relation documentation is able to convey the exact same information except with the benefit of being as free-form as necessary to get the point across. Witness the hal browser. All without adding noise to the messages at runtime thanks to the indirection of link relation URLs.

By "unspecified formats," I meant there's typically no link relation documentation that is automatically consumed by the client for the purpose of dynamic runtime client configuration.

A link relation URL is free to serve as many different content types as it likes via the usual HTTP content negotiation stuff - you're not limited to text/html. I don't know of many people who do this though - perhaps because they don't feel its worth it in practice? I think I remember a couple of HAL projects a couple of years ago that were also serving schema docs from the link rel URLs. No idea how that turned out though...

This is what would be required for the workflow use case that doesn't include this information inline. This sort of thing could certainly exist some day. Without something like this, I haven't seen a good design for this use case that doesn't involve inline actions of sorts.

I really think it would help if you could produce a concrete example of one of these mysterious "dynamic workflows" and ideally include examples of the sorts of dynamic changes you think clients should be able to adapt to and how these changes manifest themselves in the messages.

These are the benefits I need, and it's easier for both machines and developers if the information is included in the message body itself (no extra requests required).

Well, these are the affordances you think you need, because you believe it improves ease-of-use enough to be worth the additional complexity.

There is always a trade off to be made.

I often code in state machines, and I like to avoid dummy requests. If the server communicates available transitions, along with metadata describing the parameters for executing those transitions, this reduces network chatter and makes "where to go from here" crystal clear. That's beneficial for a number of reasons.

I don't know what you mean by "dummy requests" so its impossible to comment on this.

Cheers,
M

@fosdev

@zdne

So, walking into this discussion principally, if blueprint is an api design tool, imo it should support API design from an abstract to a concrete standpoint in a media-type and protocol independent fashion. That is the media-type and protocol are implementation details that follow after the the basic definition of the resource and its available affordances (ideally with a state machine in mind).

Basically, that means:

  1. I define the data semantics of my resource (what the metadata and/or attributes mean).
  2. I define the affordances associated with my resource and any requisite parameters or attributes that should be supplied with those affordances (relations, link or form).

  3. I define the statemachine associated with the resource (albeit this is more for internal documentation than external, but effectively this is important because it highlights the sets of affordances that one might experience and the conditions that must be satisfied for the affordance to be present in a response).

    At this point if I have done my job correctly I have a RESTful resource defined in its entirety and if I want to generate a machine-readable profile, say like ALPS, I can. I have as @mamund suggests, effectively defined the application domain profile associated with the resource. This to me is fundamental to API design and very important to support M2m applications.

  4. I then define the media-types that my api supports and can supply sample representations.

  5. I then define protocol specific implemenation details for the affordances (uris and uniform interface methods, headers, status codes, etc). However, from a hypermedia API standpoint

As you and I have discussed, I think the design tool should support user defined metadata in the tree that is part of the blueprint, but not part of the documentation. So, introduce into the blueprint tree a metadata tag in my description below. The metadata tag could be used anywhere in the tree and can be used to tracking media-type specific attributes as well once you get into the issue of supporting different types.

Past that, I am not necessary advocating the naming below, per se, but just using examples that I think explain the ideas.
Note: I leave any statemachine specific stuff out of the example for now.

Format: Hypermedia

# Profile Contact
This is a contact.

## Properties
  + name - The contact name
  + phone - The contact phone

## Affordances
  + self
    + metadata
        + type - safe
  + update - Updates the contact
    + metadata
      + type - idempotent
    + Parameters 
      + ...
    + Body
      + name
      + phone

## Media-types
Contact representations in different media-types.

### application/json
### application/hal+json

## HTTP

### contact/{uuid}
  + self
    + GET 
      + Headers ...
      + Response 200

  + update
    + PUT
      + Headers ...
      + Response 201

The API documentation associated with this might look like:

Profile Contact

This is a contact.

Properties

  • name - The contact name
  • phone - The contact phone

Relations

  • self
  • update - Updates the contact
    • Parameters
      • ...
    • Attributes
      • name
      • phone

Media-types

Contact representations in different media-types.

application/json

  {
    "name": "bob",
    "phone": "555-1212"
  }

application/hal+json

  {
    "_links": {
      "self": { "href": "..."},
      "update": { "href": "..." },
    },
    "name": "bob",
    "phone": "555-1212"
  }

HTTP

self

  • GET
    • Headers ...
    • Response 200

update

  • PUT
    • Headers ...
    • Response 201

Note the absense of the metatdata in the documentation. I think with a little creativity, much of the existing tree could be used and some of your markdown conventions aliased as necessary. It may be that you could use the format Metatdata at the to add aliases, etc. E.g. Media-type for Model (that latter which feels very RPC to me).

Also, we can put serializers into this that can generate the sample media types and even ALPS profiles (plug for @mamund).

@zdne
Owner

Personally I am really aligned with what @fosdev suggests.

However I would like to take some shortcuts initially. For example to build on HTTP protocol. Being protocol-agnostic is still a long-term goal however I would sacrifice it for clarity and simplicity. At least in the beginnings.

Also I am not sure whether the concept of tagging is necessarily related to this Issue's topic.

As we have agreed with @fosdev we will try to explore this approach in-depth. Recently, I have published a new API Blueprint Tutorial that is "built" on top of an imaginary API (blueprint). Since this imaginary API described in the blueprint is RESTful and yet close to an existing real-world I would like to take use it as an archetype API for this research.

@BRMatt BRMatt referenced this issue in balanced/balanced-api
Merged

Introducing Cukes #431

98 of 98 tasks complete
@steveklabnik

I'm just adding my :+1: here, am interested to see how it turns out.

@glennblock

OK, weighing in extremely late (kudos to @zzen for point this out).

First this is a great discussion and even more importantly it has brought in many of the people who really care about Hypermedia who have been giving thought leadership on the topic, so thank you! I also would to like see things move forward here as I've been on my own quest for Hypermedia and quest for educating on Hypermedia. (http://shop.oreilly.com/product/0636920026617.do)

Here are a few of my own thoughts that I'll throw into the mix:

  1. Having multiple options is good. The space is still young and we're learning. I am a big believer in the crowd sourced natural selection that happens around OSS efforts. It will happen. If that means we end up with multiple solutions, than great, if we all naturally hone in on one, then that's also fine.

  2. Please at least support a way to express Hypermedia. I think THIS is critical. Folks are starting to get on board both big and small with bringing Hypermedia into their designs. Not because they attend the church of Fielding, but because they see value. Don't block them. Don't make them say, "We like Blueprint (or some other doc format) but we can't use it because our API is Hypermedia driven". Depending on the success of such format, they will either lose by having to give it up in favor of their API or lose in giving up Hypermedia.

  3. Let's keep our eyes on the prize. Let's work together to move the bar, to make it easier for people to build hypermedia systems and not get caught up on which way is better than the other.

I am a big fan of the discussion here and I think that efforts like this can play a big part in advancing the state of the art so to speak.

@zdne
Owner

 

... in the response on this encouraging discussion:

Introducing Resource Blueprint

I would like to present you the proposal of a future API Blueprint, Mark (@fosdev) and I were working on – the Resource Blueprint.

The Resource Blueprint promotes good API design focusing on API's business logic.

It's ambition is to take the API design to a whole new level, a different perspective, not to "add syntax to API Blueprint to make writing hypermedia APIs easier". Abstract from (but not obfuscate!) protocol layers. Focus on API's business. On its resources, semantics and states.

Please let us know what do you think. I believe together we can turn the odds and make APIs accessible for everyone just as Web did to the Internet (moving from Telnet to REST Web clients).

Thank you for your overwhelming support participating in this discussion!

@mamund

i assume this effort is focused primarily on the server side, right?

@zdne
Owner

@mamund

Not at all! As with API Blueprint I want to encourage "both sides" to talk and actively participate in the API design process.

@zdne
Owner

@mamund

So my aspiration here is to find a common language to talk about an API regardless most of its technical details (unless you opt to).

Personally, I think about it as a sort of Cucumber / Gherkin for APIs. A set of features describing your program's behavior in a human language without the knowledge of program's internal technology. Not necessarily (but possibly) related to testing.

@mamund

all sounds cool. i think we're all working in the same general space - much needed.

my Q has to do w/ what the Resource Blueprints can produce. it looks to me like it is describing server implementations (e.g. the state machine) so that is why i asked the Q.

@mamund

yeah - Gerkin is something i see primarily for server developers (test & doc). that's why i am asking.

@zdne
Owner

@mamund

As with Gherkin you are indeed describing and documenting behavior of your program (server in this case). However these descriptions - features serve much grater purpose. It is contract with users, specification and documentation everyone can read.

It is not the implementation you are describing it is the behavior as observed by / agreed upon with clients. Big difference.

@steveklabnik

@zdne we at @balanced are actually using Cucumber in this way, currently.

@mamund

so you see these documents as contracts w/ client apps. the state machine is part of that contract, too? as in which links will show up in which response (not counting security context as you mention in your docs), the paths between states, etc. - that's all part of the contract, right?

@steveklabnik

Yup. https://github.com/balanced/balanced-api gets run in CI against all client libraries, our backend, and every so often against production. Any additions start as a PR with a failing spec: balanced/balanced-api#496

@glennblock

I have a similar concern that I think @mamund is raising. The server-side state machine should not be IMO part of the client contract. The whole point of hypermedia is to decouple clients from the server and this would have the opposite effect!

Perhaps there could be a server-specific document which is for the implementers of the API / used by testing tools, but not public?

This would be in addition to a client-specific document which should probably focus on the hypermedia transitions that may be available at any point?

@steveklabnik

The whole point of hypermedia is to decouple clients from the server and this would have the opposite effect!

One of the problems with teaching people about how to use hypermedia is that when you give them a page like http://jsonapi.org, they don't know how to turn that into a client. So we show off various use-cases, and verify that they actually work that way. In theory, we could say "here is a list of relations and jsonapi.org, go build a client!" but we're just not there yet with general developer understanding about clients. When I say that to people, they hear, "Did you just ask me to go fuck myself?"

It's not ideal. Baby steps. We'll get there.

@mamund

@steveklabnik: interesting.

so you are, in effect, using cucumber to document your state machine, right?

@zdne, @fosdev
so, one of the things that i think would be valuable to generate from this process would be a complete state diagram. one that allows us to scan the visual space for transitions (including possible details like safety, idempotency, params) and see where changes will affect the diagram (changing params somewhere, adding new transitions, etc.)

@steveklabnik

@mamund yes, that's right.

@fosdev

@mamund We have discussed generating state machine visuals and a stubbed api that exposes the state machine. Thus, design the API and be able to play with it as a state machine.

@mamund

@fosdev: cool. getting way afield, but the oppty to draw a state diagram and (from that) generate the Resource Blueprint would be quite spectacular ;)

@kevinswiber
@fosdev

the state machine is part of that contract, too?

@mamund @glennblock IMO, the state-machine is part of the API design, but not part of the external contract. The external contract is just resource attribute and affordance definitions (that can generate a profile), media-types and entry point.

However, as a developer understanding and surfacing your state machine then drives what you implement. That is, the state-machine and protocol implementation details in the resource blueprint are internal and guide the implementation. I suspect per @steveklabnik's comments (have not looked at his example), you could generate test/acceptance criteria from the state machine.

Further, if you define the state machine, and you have a library that can understand that, you can stub the state machine and have a sample API that works like the state machine. Not committing @zdne to this anytime soon, but we have talked about how you just design the resource and the state machine and then have a stubbable API.

Cake and eating it too, would be all of the above. First step, is designing and stubbing the API, I believe. @zdne is that authority on how he sees it.

@steveklabnik

I suspect per @steveklabnik's comments (have not looked at his example), you could generate test/acceptance criteria from the state machine.

I believe you could, yes.

@steveklabnik

@mamund @glennblock IMO, the state-machine is part of the API design, but not part of the external contract. The external contract is just resource attribute and affordance definitions (that can generate a profile), media-types and entry point.

In the ideal case, yes. What I'm saying is that I don't think we're quite there yet, so we're also contracting on the machine itself.

@glennblock

@fosdev I agree it's part of the design / no question there. My concern was if it is part of the public contract. As long as it is not, I have less issue.

@fosdev

My concern was if it is part of the public contract.

@glennblock Yeah. Me too. So, based on this ongoing concern, @zdne maybe we should clarify this in the README.

@steveklabnik

If you document it, people will use it. A comment in the README isn't gonna be enough for people to not depend on it. :/

@fosdev

@steveklabnik True. So, the blueprint is a design tool. I believe in the generated API documentation view, you would not show it, but again @zdne would clarify that.

@mamund

in general, you can't stop folks from doing whatever they think they see in the design. that's always gonna happen. but it does help to make some statements about intended use, warnings about not assuming stuff, etc. that gives folks something to point to post-facto: "See, I told you not to bake that into the code!"

@glennblock
@zzen
Owner
@mamund

@glennblock: yep. again, off topic but...

the approach on ALPS is to describe only the shared knowledge between all parties (vocab and transitions) and none of the implementation details (server-side state machine, client-side input ranges, etc.). for example, each server might have their own state machine for handling microblogging, accounting, etc. each client might use one or more server-based services to create their own independent application, etc. in fact, we designed ALPS in a way that makes it really difficult to include implementation-specific information grin.

@fosdev

ALPS/profile thinking has influenced a lot of the design ideas here and one of the uses Traits metadata is be able to use it to generate profiles from the API design. Further, just defining semantic vocab and transitions is at the heart of the resource blueprint from an external documentation standpoint.

@mamund

@fosdev: yep. all sounds good.

it might be interesting to see if things shake out such that ALPS turns out to be just the interface bits and Blueprint (and/or other stuff) ends up supplying the implementation details needed to actually create servers & clients that understand that shared interface.

anyway, great to see this kind of effort here.

@mamund

@zzen:

yep - the views generated from the document by tools can limit/focus-on all sorts of things. what tools can't/shouldn't do is include stuff that's not already there. that's why i am asking Qs about intended audience, assumed use cases, etc. just trying to find the boundaries and see what could be done within them.

@zdne
Owner

@steveklabnik

Thanks for sharing @balanced API! I think this is a great take on describing & testing APIs, worth to learning from!

Now could you briefly elaborate on some of its pitfalls? Anything that is particularly complicated or just annoying using cucumber in this way? What about abstracting a bit more from HTTP as in

I should get a 201 CREATED status code

or

Then I make a GET request to /orders/:order_id

as well as the whole JSON serialization? Would that make sense? Could you see Resource Blueprint walking in these footsteps?

@mikekelly

One of the problems with teaching people about how to use hypermedia is that when you give them a page like http://jsonapi.org, they don't know how to turn that into a client. So we show off various use-cases, and verify that they actually work that way. In theory, we could say "here is a list of relations and jsonapi.org, go build a client!" but we're just not there yet with general developer understanding about clients. When I say that to people, they hear, "Did you just ask me to go fuck myself?"

It's not ideal. Baby steps. We'll get there.

@steveklabnik I don't want to be confrontational but ... :smile:... imo, this kind of thinking is a bit backwards.

We should be trying to bring the dev experience of our APIs down to a level that is basically intuitive to the average developer, rather than attempting to elevate their consciousness. Surely this is the point of "discoverable" APIs through which you can "follow your nose"?

In other words, we should be doing user-centered design on our APIs.

I think this project is interesting, and working towards executable documentation that also drives a test suite is definitely a worthy goal... But right now writing and maintaining HTML documentation pages by hand is ok for me and does not present a significant problem. In fact, I actually quite like the flexibility this affords me as opposed to being bound by the constraints of a template language.

Still, this is very cool stuff. Keep it up! :+1:

@zdne
Owner

@mamund @glennblock

I feel I should elaborate a little bit on my statement:

So my aspiration here is to find a common language to talk about an API regardless most of its technical details (unless you opt to).

I really think about the whole API Blueprint as a common ground for talking APIs. Not only as a client's documentation. The API (Resource) Blueprint is not just an API Reference you are handing to your clients. Sure, you can generate one from ti but this is not its sole purpose.

As such the term "contract" as I have used it can be applied broadly to many scenarios different scenarios. For example:

  1. Existing service's contract with many clients

    Here the decoupling matters. As mentioned, clients should ideally care just about semantics and transitions.

  2. Contract on building a service for a customer

    Here a client (customer) orders a service to be built. Expected behavior (semantic, states & transitions) is described as perceived by the client, agreed upon in the contract and build by the service authors.

@mamund

@zdne

i hear ya. i don't think i ever used a phrase like "just an API Reference you're handing your clients." in fact, i started this convo asking if this was primarily server focused, not client focused.

FWIW, i suspect creating a single format to handle both those things is going to be very tough. diff levels of detail, diff areas of responsibility/concern, etc. but i'm all for seeing this go forward to see how it shakes out.

cheers.

@zdne
Owner

@mikekelly

this kind of thinking is a bit backwards

Well yes and no. I would love everyone to be able to "follow his or her nose" but we are not there yet. And in order to get there a lot has to happen including better clients (and HAL browser goes a long way here).

In other words, we should be doing user-centered design on our APIs.

Absolutely! This is why I want to drag clients in the process of the API design!

@zdne
Owner

@mamund

No you haven't Mike, I did not mean to accuse you from it. Sorry if it felt like it.

I just wanted to point out the contract does not have to be just this reference for clients, but also between customer and contractor where description of states as perceived by the client indeed makes a sense. And in this regards you initial observation is correct – this might be focused also on a server side of things.

@zdne
Owner

@mamund

... basically it is just now I think I fully understand your initial comment :)

@mikekelly

Well yes and no. I would love everyone to be able to "follow his or her nose" but we are not there yet. And in order to get there a lot has to happen including better clients (and HAL browser goes a long way here).

For sure, browsers for the message formats can be improved.

The key issue for me is that the viability of an API design shouldn't be dependent on developers being aware of it i.e. a well designed API should accommodate developers who want to explore it using curl or write their client using jQuery.ajax and "raw" json traversal.

That is why I think it's backwards to be concerned about whether "developers know how to write good clients to our well designed APIs"; it's a contradiction in terms.

If developers being able to write 'good' clients is something you are concerned about then your API is not well designed.

@kevinswiber
@zdne
Owner

@<all>

turning the discussion back to the Resource Blueprint proposal...

So far it seem the only dispute / concern is about the section describing the state machine of an API and what are its implication on 1. API design 2. Clients.

The States section is however juste one optional piece of the Resource Blueprint. What about the rest?

Do you find describing resource semantics and available affordances in human readable language while sweeping the HTTP and Media Type layers under the carpet appealing or does it offends you?

Assuming a reasonably good default HTTP / HAL / Siren "translation" exists could this contribute to overall better, easier and more accessible API designs?

@steveklabnik

@zdne

Now could you briefly elaborate on some of its pitfalls?

Well, we're doing outside-in development, so I write the cukes, they fail, we implement the API, the cukes pass, and we're good. But since the cukes and the tests are in two different projects, I have to merge either failing tests, or code that makes the test fail until the test PR is merged. This isn't an actual problem since the only failures are on new behavior, but I like being all green all the time, and the few minutes in between aren't ideal.

Anything that is particularly complicated or just annoying using cucumber in this way?

Right now the step implementations are a mess because I focused on getting it done above all else. There's lots of shared mutable state, which is gross. I need to go back and do some refactoring work. This only got moved to production a day or two ago.

What about abstracting a bit more from HTTP as in

There's actually some comments on the original PR from the authors of cucumber on this topic: balanced/balanced-api#431 (comment) and the few after.

@steveklabnik

@mikekelley

We should be trying to bring the dev experience of our APIs down to a level that is basically intuitive to the average developer, rather than attempting to elevate their consciousness. Surely this is the point of "discoverable" APIs through which you can "follow your nose"?

"following links in an API" is not yet intuitive to the average developer. More education is needed first.

If developers being able to write 'good' clients is something you are concerned about then your API is not well designed.

As a designer, it's my job to enable people to use the thing I've built. If they can't use it, that's my failing, not theirs.

Currently, "here's a media type and some link relations" is, to the average developer, a failure. I hope we can get there soon. But we all have a lot of work to do.

@kevinswiber

@zdne

The States section is however juste one optional piece of the Resource Blueprint. What about the rest?

Do you find describing resource semantics and available affordances in human readable language while sweeping the HTTP and Media Type layers under the carpet appealing or does it offends you?

Assuming a reasonably good default HTTP / HAL / Siren "translation" exists could this contribute to overall better, easier and more accessible API designs?

I think this is good stuff.

I'll say I share the same concerns as Mike A. and Glenn regarding the state machine. Now I'll move on. :)

  1. Is there a way to mark Attributes as optional?
  2. For Embedded Entities, how would you document cases when the list of Attributes (for instance) differs between what's available as an item in a collection versus what's available in the item's full representation?

That's all I got for now. Still noodling.

@zdne zdne referenced this issue
Closed

Embedded Assets #29

@zdne
Owner

As a designer, it's my job to enable people to use the thing I've built. If they can't use it, that's my failing, not theirs.

Absolutely love this point

@zdne
Owner

@kevinswiber

  1. Is there a way to mark Attributes as optional Is this meant as "optional in the resource representation" or "optional in attribute of the resource"?

For the former you can override the Media Type completely in the media type section. Marking it as optional could drive the default Media Type configuration.

Not sure about the later. Is it needed?

  1. For Embedded Entities, how would you document cases when the list of Attributes (for instance) differs between what's available as an item in a collection versus what's available in the item's full representation?

Using proposed referencing syntax.

This would pick whole Gist resource as defined elsewhere

+ items (array)
    + Embedded Entities
        + [Gist][]

While something like

+ items (array)
    + Embedded Entities
        + [Gist.show][]
        + [Gist.id][]
        + [Gist.description][]

Would embed just the show affordance and id and description attributes. Note I am using the same syntax for referencing both attributes and affordances. Not sure yet if that would be OK but feels natural to me (affordances and attributes are just qualities of a resource right?).

@mikekelly

As a designer, it's my job to enable people to use the thing I've built. If they can't use it, that's my failing, not theirs.

Absolutely love this point

Personally I just take that as a given. Nobody argued otherwise.

The problem is what is the right way to approach the "job of enabling people to use the thing you've built"? In my opinion, stating that "more education is needed first" is a huge smell that you are not heading in the right direction.

@zdne
Owner

@mikekelly

"more education is needed first" is a huge smell that you are not heading in the right direction.

What do you think should be done? How can we "sell" these design concepts better?

@smizell
Owner

One thing I've been playing with lately is starting with HTML as the base representation of a resource and then translating that to other hypermedia types. In doing this, you markup the resource using HTML+RDFa (or Microdata, whatever works) along with links and forms for state changes, and then pull that data out of the HTML, either server-side or client-side. Here is a very crude, incomplete example that converts HTML+RDFa to HAL+JSON.

My question is, would starting with HTML as your media type allow you to do a lot of this described here? If you used RDFa along with links and forms, couldn't you translate it to pretty much any other hypermedia type? If so, you just made your browser into a browser for any media type you're working with. It would be the ultimate HAL browser or Collection+JSON browser.

Also, could you then generate something like Blueprint or ALPS from that HTML with special crawler of your HTML API that doubled as tests for your API? Your API would not only be discoverable, but self-documenting. Developers could follow their nose through your API using their own browser.

I see other benefits as well (such as being great for search engine crawlers and noscript users), but I'll leave it at that for now. Just a thought!

@mikekelly

What do you think should be done? How can we "sell" these design concepts better?

Better designs, basically. It shouldn't have to be so complicated that it requires lengthy specifications, books, blog posts, or github threads.

@fosdev

How can we "sell" these design concepts better?

"more education is needed first" is a huge smell that you are not heading in the right direction.

Better designs, basically.

@mikekelly So, they simply pull a better design out of their hat because they don't understand design concepts and have not educated themselves on it Hmmm. What am I missing? I think the aspiration here is just enough to help one understand and come up with better hypermedia API designs (and some advanced functionality for the advanced). If you think it is missing that, I think @zdne question was what do you think could change to accomplish that better in what is being done here.

@mattupstate

@zdne After looking over the Gists example for the Resource Blueprint I feel like I'm missing something. Gists Resource specifies a state transition to a navigation state. I don't see this anywhere in the blueprint. Where is this state?

@fosdev

@mattupstate We did some editing and missed that one. Will fix. Should be selection state.

@zdne
Owner

@smizell

Do you see another benefit to using HTML as a hypermedia type besides the navigation in browser? Is designing per-resource media types in HTML and chaining them together the API design process you would use?

What about making it other way around. Transport from HAL (or SIREN) to HTML5. Sort of HAL+HTML. Do it over a proxy and you can surf a hypermedia API in your browser...

Also note API Blueprint is not a media type for hypermedia APIs. But technically, yes, you could compose a blueprint from browsing an existing API (similar to cURL trace parser).

@smizell
Owner

I think the big benefits would come for applications that are meant to be used in the browser, while it seems like it could help in educating developers on how to design hypermedia as has been discussed. Web developers understand HTML links and forms in the browser, whereas they may not iniitally understand hypermedia in JSON that they have to consume in some separate application.

Additionally:

  • If you built in HTML first, you have something you can serve up for crawlers and those with noscript. A lot of single page apps build a separate HTML site after the fact that is usually limited. Plus, Google likes RDFa and Microdata.
  • HTML would let you express links and form controls, whereas some media types like HAL don't normally include the forms (though you could add it). You can go from the greater to lesser, but not the other way (unless that information is available in some parseable document like Blueprint).
  • Likewise, using something like RDFa or Microdata allows you to include data in your HTML that is independent of the layout of the document, which means you design the document as you see fit, and it doesn't break the translators to HAL, Siren, etc. For instance, if someone is building a mobile-first site, they build their site as normal and mark it up with say RDFa, and then they can translate that to whatever other media type of that resource they want to serve from the same URI automatically. Most JSON media types require strictly adhering to the document format. This all means, with HTML, you can be more expressive with your resource (not just with forms as mentioned).
  • If there are really good libraries out there that convert HTML+RDFa to HAL or whatever, you're just a crawler away from being able to automating a good portion of your link relations for any media type that can be converted from RDFa. I look at something like haltalk and see the link relations there, and it seems those could be definitely automatically generated from HTML, but not sure if that could be done directly from HAL because of the lack of forms/actions in the responses. Is this where Blueprint would fit in?

So it seems like HTML allows you to be more expressive, even if you are just making your API browseable with it. With that said, it has its limitations, such as only currently supporting GET and POST, so you'd have to hack it with _method inputs like Rails.

Is designing per-resource media types in HTML and chaining them together the API design process you would use?

I'm not sure yet, but I'm thinking about it! I'm trying to build some tools to see if it simplifies my design/development process, such as the HTML-to-HAL one above. It does seem like it simplifies things when all you have to build is HTML for your API and the rest is automatically done for you.

Edit: just to clarify, I don't guess you'd have a bunch of resource-specific media types, but rather use HTML for everything with RDFa to convey your domain semantics.

@glennblock

@smizell Jon Moore would definitely love this idea ;-)

I agree you "can" use HTML for anything as it has affordances built in and is very flexible, however it comes at a cost. I think it's real value is I can hand it off to any browser and it can render it. But....You have to work within the existing facilities to express what you want, and that to me is a big downside. If I create a profile for example on JSON, I can tailor JSON it so it really fits my domain and is very obvious as JSON at it's lowest level is just a lightweight way to express dictionaries and lists. If I go down the HTML route I have to work within the constraints, use the attributes it provides like name here, class there. If I see an <LI> element and say yeah that's a list, but it's really a list of contacts because it has a class="contacts" attribute.

Also I find that consuming HTML in a JS application is less than ideal.

@smizell
Owner

@glennblock I'm actually working on a library called halpert to address these things you mention, and there is a jQuery version to use for doing this in the browser. This allows you to mark up your resource content using HTML+RDFa, which is much more expressive than using classes/ids, and then parse it to HAL+JSON. Here is an example (view the source), with the parsed HAL+JSON on the right that is derived from the HTML document.

I didn't mean to hijack this thread with HTML talk, but my original intent was to mention that starting with HTML as your hypermedia type allows you to address a lot of the issues mentioned higher up, even things like increasing education and awareness on hypermedia design.

@drewzboto

I'm totally late to the party, but thank you for the rich discussions so far. Definitely mirrors discussions happening as part of our struggles to define APIs internally, as part of improving both client-side and server-side developer experiences....

Just reviewing the Resource Blueprint. I think it's a step in the right direction, which I haven't seen any other similar initiatives take yet. One thing that I'm either confused or concerned (confuscerned ?) about:

Includes transitions in different states (business rules) and the conditions (permissions) for their inclusion in a response.

Transition to different states - from the client side this makes sense. Here are the different actions you can possibly undertake, and should look out for as part of a response. The conditions to me are part of the underlying business logic, which I’m trying to shield the client from with an API. Why this affordance appears in the response shouldn’t be a client side concern. From a server-side perspective, the conditions can and should evolve as my system grows, and I should be free to do so without revisiting my resource definition, as I don't think it breaks my client-side contract.

From a practical point of view:

  • I have a shopping cart
  • I have a checkout affordance, which appears if I fill out all my cart information as part of state transitions
  • As a system, I need to add in a new condition which hides the checkout action if terms and conditions have not been agreed to by a given state transition. Is this something I need to document here as a condition? It’s logic behind the API, which may not even be a system I control.
@fosdev

@drewzboto If you look at the use of Conditions in say the create transition in the sample API, you will see it is not an affordance. Rather, as part of the API design process, you need to ask questions about conditions for inclusion.

Whether it is business logic (does the transition even exist) or permissions (the transition is available in the context of the user, role, etc) is all transparent to the client and is encapsulated in the API implementation as you note. Not everything in the blueprint will roll over to the human readable documentation generated by it. It is a good question whether conditions should. Presumably somewhere if there are permissions that should be set up, you would want to document their applicability. Not sure that would be via Conditions.

Given the blueprint is itself a design tool, thinking about permissions separate from the existence of a transition in a state is a useful abstraction, and thus the point of its inclusion. @zdne may have other thoughts.

@zdne
Owner

@drewzboto

I'm totally late to the party

No, I would say the party is just about to start – we are only at the dawn of the "API design" age.

Is this something I need to document ...

No. The resource blueprint is a permissive language. The minimum reasonable blueprint would include just a resource with one attribute, perhaps one affordance (other than "self").

Documenting conditions clearly depends on the API designer. Should I document a condition that controls whether I can "create" a resource? Probably yes. Should I document a condition that controls whether I can perform a "checkout"? I would say no if it is obvious (e.g. cart is empty clearly implies checkout does not make sense). But again it is up to the designer(s) to decide. The option is here.

As for rendered documentation – it is a good question whether and how it should capture the conditions. My feeling is it should. Important thing here is that the blueprint shouldn't be written in the respect "how rendered documentation looks" but with the focus on the particular API design and the use of blueprint in the API lifecycle process. Rendered documentation is just a tool using it, and there are hopefully many, not just the one from Apiary.

@Baggz Baggz referenced this issue from a commit
@Baggz Baggz Enable experiment #13 53b7d27
@stefanpearson

Are there any updates to this? Would be great to see a more efficient way of using hypermedia links/embeds in blueprint soon!

@zdne
Owner

@stefanpearson Yes, we are making quite a progress towards Resource Blueprint with the MSON.

MSON is the building stone for Resource Blueprint as it will enables the description of Resource attributes (semantic descriptors) and also of the affordances.

I also plan to update Resource Blueprint concept soon. I will keep you posted in this thread.

@stefanpearson

That's good news :), I'm having to do some very questionable pre-processing to get around this!

@zdne
Owner

I'm having to do some very questionable pre-processing to get around this!

@stefanpearson can you share more details on this? perhaps there is something we can learn from it

@stefanpearson

Well I haven't implemented it yet, but my end-goal at the moment is to:

  • Have a clean model for each resource (not including the hypermedia)
  • Have a schema for each model (not including the hypermedia)
  • Set up link relationships and embed records (embeds can also have nested links and embeds). These references will pull in the respective resource model and schema, to create the actual representation of a payload.

It's worth noting that I'm also using the resource/action schemas (JSON schema) to validate incoming request bodies, and validate outgoing response bodies within my tests. In lieu of this, the JSON schema output needs to actually include and represent the relevant hypermedia.

At the moment, as a workaround, my immediate thought is to:

  • define relationships (probably outside of the blueprint)
  • parse the blueprint to get the JSON representation
  • use a templating engine to inject the embeds/links (thus creating a valid 1A blueprint, albeit with buckets of repetition)
  • parse it again!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.