Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query for Post List #4

Open
EdwardHinkle opened this issue Aug 9, 2018 · 33 comments

Comments

Projects
None yet
7 participants
@EdwardHinkle
Copy link

commented Aug 9, 2018

Discussion for adding a way to query a list of posts from the Micropub endpoint (https://indieweb.org/Micropub-extensions#Query_for_Post_List)

Below are some thoughts from the brainstorming page.

Use Cases

  • [[headless_CMS|headless cms]] / Separation of data storage and data display
  • Private note taking / journaling
  • Find and update existing posts eg. change a "want to read" to "read" after complete
  • Full content management UIs
  • Find and edit draft / hidden posts
  • Synchronizing various documents eg. bookmarks
  • Discover if a user has liked / replied to a specific page

Query Parameters

In order to achieve the above use cases there would need to be a robust query system in place. It would need to be able to query for:

  • url (this filters and returns the single post with that url. This is how q=source currently functions)
  • posts based on post type (similar to #1)
  • pages of posts with defined limits
  • posts with specific properties
  • posts with combinations of properties
  • posts without certain properties
  • posts with properties that match a certain value (eg, all posts that have in-reply-to == 'example.com')

Other Considerations

  • Any returned item in a list would need to include a url property so that micropub clients could use the url for update and delete requests etc.
  • Would require one or more scopes
@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Aug 9, 2018

Here are the notes about IndieWeb implementation from the brainstorming page:

  • j4y_funabashi implemented an extension to the q=source query on my micropub endpoint, if no url is provided it returns a list of posts. Not yet paginated but that is the plan.
  • eddiehinkle implemented the q=source query extension.
    • When no url is provided, it returns a list of posts.
    • It doesn't support pagination yet, but currently returns all posts within the last month
    • It does support a post-type filter that will filter out what posts are returned based on the post type discovery, similar to #1
  • grantcodes implemented an experimental extension that uses the mongodb query syntax since his posts are stored in MongoDB
@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Aug 9, 2018

@cleverdevil Post List Query discussion has moved here, for when you want to work on integrating into Known

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Aug 9, 2018

It has been discussing that using paging should probably be similar to Microsub (https://indieweb.org/Microsub-spec#Example_Paging_Workflow) where the server provides both after and before strings for pagination. The server will then return results as appropriate. When you reach the point that there are no more results available, the results don't contain an after string.

The pagination tokens are opaque to the client, so they can be anything the server wants: page numbers, date string, post ids, etc.

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Aug 9, 2018

It probably makes sense to include the limit parameter from Microsub as well. So that would put us at:

https://example.tld/micropub?q=source

  • &post-type=type - filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
  • &after=pagination token - returns older posts
  • &before=pagination token - returns newer posts
  • &limit=number - limits the returned results by the number provided

As already included in the original spec:

  • &url=POST URL - returns a single post that matches this url
@grantcodes

This comment has been minimized.

Copy link

commented Aug 10, 2018

I think the specific property queries might be more difficult to map into the url parameters.

They could potentially follow a similar format to the micropub update spec

In particular like the delete parameter which can either specify just the property name to delete or the value to delete.

In my mind there needs to be 2 types of queries on properties at least:

  1. A specific property exists
  2. A specific property contains a given value - eg. post-status = draft
    This could get more complex, if you wanted to match an entire array (eg. category = ['foo', 'bar']) or just contains the given parameter

There are also a number of others that may be worth having, but I am not so sure are as vital:

  1. A property does not exists / is empty
  2. A property value does not equal a given value
  3. A property value is less than, greater than etc. compared to a given value
@grantcodes

This comment has been minimized.

Copy link

commented Aug 10, 2018

Another one worth considering would be a sort or order parameter. I can imagine clients wanting to either retrieve posts in chronological or reverse chronological order at a minimum.

And potentially even ordering by other properties, eg. Alphabetical by post title. Although I can't think of a solid use case for that right now.

@grantcodes

This comment has been minimized.

Copy link

commented Aug 12, 2018

I've updated my implementation so I support a bunch of new features expanding on what @EdwardHinkle already mentioned

When using q=source there is now a lot of other parameters available:

  • post-type=type - Filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
  • after=pagination token - Returns older posts (I just use page numbers for now)
  • before=pagination token - Returns newer posts (I just use page numbers for now)
  • limit=number - Limits the returned results by the number provided
  • order=reverse - Returns posts in a reverse order
  • exists[]=property name - Returns posts that include the given properties
  • not-exists[]=property name - Returns posts that do not include the given properties
  • property-property-name=value - Returns posts that contain the given property value e.g. property-post-status=draft
@cleverdevil

This comment has been minimized.

Copy link

commented Sep 6, 2018

FYI, I have a local branch of Known that supports:

  • q=source
  • &url=
  • &limit=
  • &offset=

I'm closely watching this thread, and if we can come to a consensus, I should be able to have a fully compliant implementation for Known in short order.

Things that need to be decided, from what I can tell:

  • What's the official approach to pagination?
  • Of @grantcodes' and @EdwardHinkle's additional query parameters, which should be adopted?

I will say that the &post-type parameter would be a bit difficult to implement in Known at this stage, because Known doesn't currently have any such notion. That said, a mechanism could be created that would map Known "Entity" types to post-types from the Post Type Discovery spec. I'd likely need direction from @benwerd and @mapkyca on how to do it.

@grantcodes

This comment has been minimized.

Copy link

commented Sep 9, 2018

Can someone write down here a valid use case for post type queries? I'm sure there were some but I can't remember why they'd be needed if property queries are supported

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Sep 9, 2018

Here is a link to the previous reasons cited: https://chat.indieweb.org/dev/2018-08-09#t1533837539815600

To summarize, there are instances where just a single property existing does not define some more complicated post types. So it means you either have to define complicated requirements in URL parameters (including that certain properties DON’T exist) or you use the PTD by using a post-type parameter.

Some examples:

  • both Podcasts, Articles and Issues have a Title. So how do you get all articles but not podcast episodes or issues when they all have titles?
  • Determining the difference between Video and Photo posts because Videos often have a photo property.
  • Using PTD protects against new experimental post types getting returned erroneously because it happens to have the same property as another post type.
@swentel

This comment has been minimized.

Copy link

commented Sep 16, 2018

I like the post-type filter, and I guess in combination with the post-type discovery, we can either expose them or not in the micropub clients. In the case of the Drupal module: I map every micropub post to a different content type in Drupal. When a q=source would come in with a post-type filter, it would be very easy to filter out, so I'm +1 on that filter :)

I'm going to experiment with this the next week to add this to the drupal module and into indigenous for android. Since I've added basic update post support, having a list to quickly edit (instead of having to copy over urls and such), a very simple list would make my life extremely easy!

@swentel

This comment has been minimized.

Copy link

commented Sep 21, 2018

So the post list and post-type filter is in the latest release of the indigenous android client and drupal module. The post list listens to the post-type discovery as well, so if none is found, filtering is not possible at the moment. I'll gradually add more filter options (like limit) and paging in the next week.

@grantcodes

This comment has been minimized.

Copy link

commented Oct 15, 2018

So now that this is becoming more widely spread, I am starting to have issues with implied post types and micropub queries. Mostly around whether something should be a specific post type or if it should be any post type with a specific property.

My 2 main use cases, (which I think are totally valid, and either in wide spread use or wanted):

  1. Drafts - Obviously there is post-status = draft But drafts are not defined by post type discovery at the moment. So at the moment to get a list of drafts there would either need to be a post-type=draft query or the client and server would both need to support property-post-status=draft.
  2. Private journal entries - Again there is no post type defined for journals, and this time no property that is specific to journals. Personally I just make a note with post-status=private and category=journal. If I wanted to expose these to a micropub client I would likely need to make my own custom post type.

Having written these out I think my main issue is around the creation of custom post types and whether they should be actual post types, or just properties and how that works with micropub queries

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Oct 15, 2018

@grantcodes it looks like half your post is missing because you only mention 1 use case

@grantcodes

This comment has been minimized.

Copy link

commented Oct 15, 2018

@EdwardHinkle Yeah, published early by accident, updated now.

@swentel

This comment has been minimized.

Copy link

commented Oct 15, 2018

What if we do something like this as query params for properties:

  • property-name=value
  • property-value=value

That would make sure we don't end up with a dozen of property-{property-name} options for the query.

@grantcodes

This comment has been minimized.

Copy link

commented Oct 15, 2018

@swentel how would you then do multiple values at once?

@swentel

This comment has been minimized.

Copy link

commented Oct 15, 2018

Sending an array then: property-name[0]=post-status&property-value[0]=draft&property-name[1]=name&property-value[1]=title

Which makes it probably a bit less readable, but still remains flexible, at least imo.

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Oct 15, 2018

I do think post-status should be a property query, and I think something like a journal is best handled as a category property query. So I think adding property-post-status and property-category (not tied to those names) are two properties that would be very widely used and wouldn’t be confused as “conflicting with post types”

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Oct 15, 2018

I don’t think the post-type filter should be the only filter, I just think it’s the simplest filter when you are trying to get a post type.

That said, I think it’s okay for a specialized Micropub app to support an experimental post type and for servers that want that to support it as well. For example: Teacup would query Ate or Drank posts which aren’t part of Post-Type Discovery. But I think general purpose Micropub (Quill, Indigenous, etc) should probably stick to mostly the widely accepted post-types.

@EdwardHinkle

This comment has been minimized.

Copy link
Author

commented Oct 15, 2018

Hmmm interesting @swentel. I can’t tell how I feel about that. Good or bad lol. A part of me thinks it’s good and the other part cringes a little.

@aaronpk With your API experience what do you think about the properties as far as being defined or flexible like @swentel suggested?

@grantcodes

This comment has been minimized.

Copy link

commented Oct 15, 2018

@swentel That handles multiple values of different properties, but what about multiple values of a single property?

property-name[0]=category&property-value[0]=foo&property-name[1]=category&property-value[1]=bar

vs

property-category[0]=foo&property-category[1]=bar

In my case I used the property- prefix because I found it easier to handle on the server. As I look for parameters prefixed with property- then I check if they are an array or string and handle them appropriately.

@swentel

This comment has been minimized.

Copy link

commented Oct 15, 2018

Oh right, multiple values, good catch. Mmm, I'd say, use comma separated values then. I can see myself filtering on two post-types as well, so post-type=note,article should work (even though Indigenous filter screen only allows you to select a single value though atm).

Note: I'm totally fine with property-{property-name} as well. It probably makes the spec clearer in the end anyway.

@aaronpk

This comment has been minimized.

Copy link
Member

commented Oct 15, 2018

I still have some catching up to do in this thread, but just want to say -1 for comma-separated values. That adds a lot of complexity into the spec. Instead, let's use the existing mechanism that Micropub has for multiple values, which is to use the array syntax instead: prop[]=val1&prop[]=val2. Also note that Micropub explicitly defines the array syntax to be the square brackets with no offset number. https://www.w3.org/TR/micropub/#form-encoded-and-multipart-requests

@dshanske

This comment has been minimized.

Copy link
Member

commented Nov 17, 2018

The WordPress Micropub endpoint now supports q=source with an optional limit parameter to return the last x number of posts. Likely implement paging in a future version.

@manton

This comment has been minimized.

Copy link

commented Mar 24, 2019

Sounds like paging is still an open question. Because Microsub was mentioned a few times, why not just copy the Microsub spec exactly: wrap the response with items and paging fields and use JF2 syntax to simplify everything. (Obviously that's a breaking change with existing implementations, but it doesn't seem like this is widely deployed yet except in home-grown solutions.)

If that's not possible, I'd probably be in favor of simple limit and offset params. Just wondering if this is an opportunity to simplify some of the list/get/edit/delete parts of Micropub, though.

@swentel

This comment has been minimized.

Copy link

commented Mar 24, 2019

The Drupal module q=source call implements paging like @manton describes, wrapping the posts in 'items'. It also supports filtering via 'limit' and 'post-type' GET params. This is all supported in Indigenous for Android too.

@manton

This comment has been minimized.

Copy link

commented Mar 24, 2019

Thanks @swentel! And does it use JF2 like Microsub, or the more verbose MF2-style JSON syntax with a properties field and arrays for values? (Assuming the latter but want to be sure.)

@swentel

This comment has been minimized.

Copy link

commented Mar 24, 2019

@manton The simple microsub version. I think Wordpress does the same, @dshanske should be able to confirm that.

@dshanske

This comment has been minimized.

Copy link
Member

commented Mar 24, 2019

Only limit so far for me, but paging is on the horizon

@grantcodes

This comment has been minimized.

Copy link

commented Mar 25, 2019

I don't think there should be any use of JF2 anywhere, it is not used anywhere else in the spec, you post mf2 and receive mf2 via source=url queries so might as well stick with that.

@manton

This comment has been minimized.

Copy link

commented Mar 25, 2019

I've been testing this on Micro.blog using MF2-style responses and plan to roll it out tomorrow. I wonder if a goal for IndieWeb Summit should be to finalize some of these extensions and update the spec with more examples? (I'm going to document this in Micro.blog's help as well, because I think it can be a little confusing to implement for developers who are not familiar with the IndieWeb already.)

@manton

This comment has been minimized.

Copy link

commented Mar 28, 2019

Support for q=source (and editing and deleting posts) has been deployed on Micro.blog. It uses the MF2-style JSON as described on the wiki for the response. Thanks y'all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.