proposal: http: FileServer: add json content type for directory serving

[posener]

Proposal for the http.FileServer:

In case that the Accept request header contains application/json, and the path leads to a directory, return the content of a directory as JSON.

Reason: add a bot an easier way to parse directory content.

Similar to: NGINX has autoindex_format option that has a json argument which makes NGINX return directory listings in a json format. The returned format is a list of maps, each map contains name, type (directory/file), mtime (modify time) and files also contain size in bytes, for example:

[
{ "name":"c", "type":"directory", "mtime":"Sun, 21 Jan 2018 08:13:49 GMT" },
{ "name":"xxx", "type":"file", "mtime":"Sun, 21 Jan 2018 08:13:49 GMT", "size":17 }
]

Doesn't mean that this is a good representation, we can debate if we want to adopt it.

[andybons]

This proposal is incomplete without context on why this feature solves a problem for our users.

What is the use-case for such a feature? What problem is it solving that you’ve hit in the past?

[bradfitz]

Also, which JSON format? Is this a standard? Do other servers do this?

[posener]

@andybons @bradfitz sorry about that, updated the issue with relevant information.
I don't know about a standard, I know that NGINX has this option.

I thought it would be a nice-to-have, can make the parsing of HTML content redundant in some use cases. I looked in the code, doesn't look to me like a big change.

[andybons]

I looked in the code, doesn't look to me like a big change.

Quoting Brad from #23782 (comment):

FWIW, the cost of this change isn't measured in characters or lines modified. It's a question of how much ongoing maintenance cost there is keeping this a stable interface for people to use long-term and whether that's even the API we want people using.

I thought it would be a nice-to-have, can make the parsing of HTML content redundant in some use cases.

You mention “add a bot an easier way to parse directory content“ but we need more detail. Why isn’t it sufficient to create a handler that walks the directory itself and returns the JSON format you want? Why do you need this specifically? Is this a common thing that people use? Noting the existence in nginx is helpful but we don’t know what it’s used for or if it’s popular. What do they/you use it for?

[posener]

Nice quote :-) I agree.

You mention “add a bot an easier way to parse directory content“ but we need more detail. Why isn’t it sufficient to create a handler that walks the directory itself and returns the JSON format you want?

What do you mean "create a handler"? some sort of copy of the FileServer function? We could create a 3rd party handler that does this of coarse. Parsing the HTML of the exiting FileServer in client side is possible, but it is more work, and has less information.

Why do you need this specifically? Is this a common thing that people use? Noting the existence in nginx is helpful but we don’t know what it’s used for or if it’s popular. What do they/you use it for?

As we all know, the question if a thing is common is hard to answer, I guess it is is not that common, since people will tend to parse HTML content of the existing and common implementation.

I saw the use in this JSON format in an application I worked on recently (distributed log server), and thought it will be nice to have in go. Apparently the authors of NGINX thought it would be useful too.

[rsc]

Please copy FileServer and publish this as a separate package. If it's wildly successful, we can bring that code back in. Note that your fork of FileServer need not reimplement single-file serving. It can call back into ServeFile, which is where most of the pain is.

[jasonwbarnett]

@posener I'm also interested in this. I was thinking of doing just as @rsc suggested and using nginx's c code as the primer for the first iteration. Let me know if you want to jam on it with me.

[jasonwbarnett]

It's legitimately awful code, but I have it working here: https://github.com/jasonwbarnett/go/tree/json-web-server

I will work to pull it into a small package and clean it up.

[posener]

@jasonwbarnett ,
Nice work, I have a few comments:

Please copy FileServer and publish this as a separate package. If it's wildly successful, we can bring that code back in.

Please notice that it is not going to be merged to the standard library in the near future - you should create a package - not copy the whole standard library :-)

In case that the Accept request header contains application/json, and the path leads to a directory, return the content of a directory as JSON

I think that the JSON response should be according to the Accept header of the request.

[andybons]

Hey folks,
It's great that you're working on this. Would you mind moving the discussion to @jasonwbarnett's repo?

Thank you

[jasonwbarnett]

@posener got it. will do as suggested. thanks
@andybons Posted this for continued conversation: jasonwbarnett#1

Lock this thread.