New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server & compile should have parity where possible #149

Open
drewhamlett opened this Issue Nov 7, 2013 · 16 comments

Comments

Projects
None yet
9 participants
@drewhamlett

drewhamlett commented Nov 7, 2013

From using other static site generators, I'm use to seeing articles/first.html get compiled to articles/first/index.html. Why is this not the case? Or am I missing an option.

@steverandy

This comment has been minimized.

steverandy commented Nov 9, 2013

You can structure it like articles/first/index.html

@kennethormandy

This comment has been minimized.

Collaborator

kennethormandy commented Nov 9, 2013

Hey @drewhjava, thanks very much for opening an issue. @steverandy is correct, right now the way to accomplish this is to make it an index file inside first/.

This is intentional, since the web server gives you the extra features like allowing you to access first.html at /first and harp compile is more strict. There is still some stuff being worked on around this, but it’s all described in #73 (comment).

Your issue does bring up a great reason to have harp compile --strict in addition to harp server --strict. Hypothetically, harp compile could could compile articles/first.ejs to articles/first/index.html and harp compile --strict could compile articles.first.ejs to articles/first.html.

@drewhamlett

This comment has been minimized.

drewhamlett commented Nov 9, 2013

Thanks @steverandy and @kennethormandy. I went ahead and structured it like steverandy mentioned, although I do like what I hear with the strict mode on and off.

@drewhamlett drewhamlett closed this Nov 9, 2013

@edrex

This comment has been minimized.

Contributor

edrex commented Nov 11, 2013

This is a bit of a pain point for me as well.

  • If I want to serve an index page at articles/ the source file needs to be articles/index.ejs.
  • My layout depends on title, etc metadata field so index file has to be present in articles/_data.json
  • Therefore I have to filter out the index whenever I'm listing articles, which is inelegant (I like putting typed data in buckets, and the index is not of the same type as the articles themselves).

I do wish that, in the case that both articles/ and articles.ejs exist, articles.ejs would be compiled to articles/index.html.

@edrex

This comment has been minimized.

Contributor

edrex commented Nov 11, 2013

Alternatively, if an articles key in the top level _data.json were applied to articles/index.ejs, that would also resolve the "mixing of types" , and it seems less magical. It would also make the directories more tidy.

It seems like there are a bunch of issue numbers around these topics, should I post my comments elsewhere?

@edrex

This comment has been minimized.

Contributor

edrex commented Nov 13, 2013

It seems like the output of compile should be equivalent to the behavior of server. Otherwise, you are really maintaining two separate, incompatible products.

@kennethormandy

This comment has been minimized.

Collaborator

kennethormandy commented Nov 13, 2013

@edrex I think I agree when that’s possible, but since Harp is adding features that other web servers might not have, obviously the compiled output can only do so much. But where parity is possible, I think that makes sense.

This is where a --strict mode would help. Right now, compile is basically like compile --strict already, but this could be changed so harp compile matches harp server and then harp server --strict and harp compile --strict match. I’ll re-open this issue so this doesn’t get lost.

@sintaxi

This comment has been minimized.

Owner

sintaxi commented Nov 13, 2013

If I want to serve an index page at articles/ the source file needs to be articles/index.ejs

stands to reason no? though harp will soon redirect /articles/ to /articles if you have an articles.ejs file instead.

My layout depends on title, etc metadata field so index file has to be present in articles/_data.json

have you tried adding title to the harp.json or setting a fallback title in your layout so you don't need it in your _data.json?

Alternatively, if an articles key in the top level _data.json were applied to articles/index.ejs, that would also resolve the "mixing of types" , and it seems less magical. It would also make the directories more tidy.

This is a pretty interesting suggestion. Not exactly sure what the side affects of this behaviour would be (both from a technical perspective and in terms of expected behaviour).

It seems like the output of compile should be equivalent to the behavior of server. Otherwise, you are really maintaining two separate, incompatible products.

The job of harp compile is to consolidate all the assets to HTML/CSS/JS so you can build apps with harp and server with other web servers. Any harp compiled app should behave exactly the same as a non-compiled app when served by Harp. Harp compile shouldn't try to make up for all the things that harp server can do with other web servers. Though it would be interesting to try.

The Harp server is indifferent about the trailing slash by doing intelligent redirects. letting you put the file where you want it articles.ejs or articles/index.ejs. Other web servers don't typically do this but I feel that is a bug in other web servers. Its hard for harp compile to make up for the deficiencies of other web servers.

@kennethormandy there are no plans for a compile --strict only a server --strict so that harp mimics the behaviour of typical web servers forcing you to build your app in a way that makes them happy.

@kennethormandy

This comment has been minimized.

Collaborator

kennethormandy commented Nov 13, 2013

Fair enough, I suppose harp server --strict is meant to match the use case of compile anyways, so giving compile another option is convoluted. For people using GitHub Pages and other things like that, though, I can see why it would nice if about.ejs compiled to about/index.html instead of about.html. That is one place where harp compile can make of for those deficiencies.

@edrex

This comment has been minimized.

Contributor

edrex commented Nov 13, 2013

Summarizing the current state below, please point out any incorrect/contested points.

Use Cases

Directory index pages (original subject of this issue)

Recommended directory structure:

articles/
  _data.json
  index.jade
  article1.md
  article2.md
  <etc>

Caveats

  • You must always filter out slug = "index" when iterating over public.articles.data (including in articles/index.jade)

Non-index pages omitting the .html suffix.

Recommended directory structure:

my-article/
  _data.json
  index.jade
  <etc>

Caveats

  • my-article won't be a key in public.data, so iterating over articles will be tricky.
  • The recommended directory structure is rather verbose/boilerplate-y for lots of articles

Possible improvements

Copy public.data.foo.* to foo/index.jade local scope if public.foo.data.index isn't defined

Would ease several of the caveats:

  • Directory index case:
    • No need to filter out slug == "index"
  • Non-index omitting .html suffix case:
    • my-article would show up in public.data, making indexes easier to build
    • No need for my-article/_data.json

While introducing a new one:

  • It wouldn't be possible to generate an index for a directory containing both foo/index.html and bar.html documents, since they have different URLs but there would be no way to distinguish them given the metadata. This could be worked around though by adding a metadata field like is_directory: true and using that to decide how to generate the URL.

Always transform foo.<jade|erb> to foo/index.html

It's pretty subjective whether the various effects of this change would be positive or negative, so I will just list them:

  • Standard webservers will serve the file at foo/
  • If you want the extension you can name your file foo.html.jade

Another possibility would be to introduce a global flag to control the filename transformation, but that would seem to go against the strong convention-over-configuration design goal.

@edrex

This comment has been minimized.

Contributor

edrex commented Nov 15, 2013

Yesterday I deployed my site to S3, and came up with a short script to get the URL style I wanted (no .html or / suffix).

Since each static deployment scenario (S3, Apache, Nginx, etc) has it's own way of dealing with clean URLs, it may be best to have harp compile opt out of trying to handle clean URLs, letting it be handled in deployment instead.

While the idea of applying public.data.foo to public/foo/index.jade may have some value, it seems like simply fixing the bug that prevents public/foo.html from coexisting with public/foo/ in server mode so that indexes can live outside their subject directory is an uncontroversial step that may suffice.

@samsonjs

This comment has been minimized.

samsonjs commented Feb 16, 2014

I'm not sure if this is really in the spirit of Harp or not, but what about having some hooks for people to run scripts after compilation? Or perhaps a plugin (though I like the script idea better). I have pretty simple needs and it would be trivial to do something like:

for FILENAME in public/posts/*.html public/projects/*.html; do
    [[ "$FILENAME" = "index.html" ]] && continue
    DIRNAME="${FILENAME%.html}"
    mkdir -p "$DIRNAME"
    mv "$FILENAME" "$DIRNAME/index.html"
done

But perhaps there are other use cases where this falls down and harp compile should do it itself. By passing some environment variables to the post-compile scripts you might be able to go pretty far. Scripts that prove themselves could graduate up into harp eventually, but in the meantime the community could serve themselves.

edit: I guess it makes more sense to just put harp compile in a script. Read the various JSON files if necessary.

@opichon

This comment has been minimized.

opichon commented Feb 16, 2015

I can see the logic for not adding the .html extension to links during compilation, but this does mean that a harp site cannot realistically be used in a non-harp server environment such as s3, unless all links are fixed by hand in the code -- or am I missing something?

For example, if my "about" page is defined in the root (or public) folder, and referenced in the source code as a(href="/about"), then the compiled link's href attribute will remain unchanged ("/about") and will not work when the site is published to S3. Converting /public/about.jade to public/about/index.jade works, but it means having 1 directory per page, which is cumbersome. And of course this solution does not work for blog posts, which by construction are stored under the same folder; so the links to the blog posts in the blog/index page are not working in S3, since they do not have a .html extension.

So pretty much the only solution is to add the ".html" extension to each link by hand in the source code. Whether or not this is an insurmountable inconvenience is for each one to decide, but it would be good to be aware of it at first, before writing a whole site and finding out that the links don't work. More importantly, does this cause some incompatibilities when the site is run in harp server?

@sintaxi

This comment has been minimized.

Owner

sintaxi commented Feb 17, 2015

Harp handles these urls just fine. IMHO this is a bug in all OTHER web servers. The lack of sane defaults such as this in other web servers was motivation for creating harp in the first place.

@daku

This comment has been minimized.

daku commented Jan 22, 2016

The harp service handles it just fine, the problem is with 'harp compile'. Why not have 'harp compile' take an input for a root path? Then, prepend all absolute references with the root path.

@tay1orjones

This comment has been minimized.

tay1orjones commented Jun 2, 2016

There's a hacky workaround I found with using the NODE_ENV variable and a base tag - see my comment on #530

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment