Restructure project file tree and build sources

[sudoforge]

Conversation pulled from #209 (review).

[sudoforge]

Comment left by @epage on Apr 8 2017 02:44 AM UTC (source)

Reason you are removing the leading "_"? I believe those tell cobalt to ignore these folders so they don't get processed or copied over as assets?

[sudoforge]

Comment left by @bddenhartog on Apr 10 2017 19:56 PM UTC (source)

Ah, I see that you are correct - /src/cobalt.rs L23-L25. I'll add an underscore to templates, but -- off-topic -- I wonder if this is really the best way to handle this? Using a pattern for ignoring could cause problems for the user (what if I decide to add a folder called _images?); usually things like this are handled via an explicit list or from a configuration value, like the ignored property in .cobalt.yml. What are your thoughts? This conversation should probably be in its own issue.

_drafts actually shouldn't be changed here, as this PR is strictly for _layouts -> templates. I'll remove this change.

[sudoforge]

Comment left by @epage on Apr 10 2017 21:40 PM UTC (source)

I think thats a good idea to split this into another issue.

Personally, I think moving all build/cli-related files into a single .cobalt subdirectory allows us to add new tooling and features - like skeleton files - without having to worry about polluting the project space. It makes it clear what the user is responsible for maintaining within their project, and where they can go to modify how cobalt works in regards to their project.

Just to check, is your comment about a ".cobalt/" meant to be related to avoiding hiding files with a "_" prefix?

[sudoforge]

Just to check, is your comment about a ".cobalt/" meant to be related to avoiding hiding files with a "_" prefix?

Yes, my previous comment is suggesting using a .cobalt/ directory in the project root to store things used for building the project but not actually built to html themselves, like skeleton files and other things the user may want to modify (e.g. templates). In general, I think the file tree for cobalt projects could be vastly improved upon for both clarity and sanity. Something like:

my-awesome-site
|- .cobalt/
|  |- templates/
|  |  |- default.liquid
|  |  |- big-header.liquid
|  |  |- some-other-template.liquid
|  |
|  |- skel/
|  |  |- template.liquid
|  |  |- page.liquid
|  |  |- post.liquid
|
|- dist/
|  |- posts/
|  |  |- post-1.html
|  |  |- post-2.html
|  |
|  |- index.html
|  |- contact.html
|  |- portfolio.html
|
|- src/
|  |- posts/
|  |  |- post-1.md
|  |  |- post-2.md
|  |
|  |- index.liquid
|  |- contact.liquid
|  |- portfolio.liquid
|
|- .gitignore
|- cobalt.config.yml
|- README.md

Then, of course, constraining cobalt build to build *.md and *.liquid files only within src/. Right now, we build any *.md or *.liquid files found in the project directory, which includes things like README.md and any files ending with those extensions in say, node_modules/ or what-have-you, which is definitely not what we want to do.

I realize that some of the points I've touched on can be configured through .cobalt.yml, but I guess what I'm getting at is that saner defaults is a good thing, and explicit whitelists (specifying exactly where we want to find source files to build) are better than general blacklists (specifying a couple places that we don't want to build). The usage of the .cobalt/ directory is, again, simply to avoid polluting the project root with source files only used by the builder.

[epage]

RE: changing defaults

I concur that we should re-evaluate the defaults. You mentioned the problem with project files being mixed in with cobalt files. There is also the awkwardness of the destination being included in the source, requiring special work for cobalt to ignore it. As an aside, in thinking about what are the selling points of cobalt.rs, I figure performance should be one and am looking for how we can design for performance where it doesn't compromise the workflow for the user (that last part being very subjective).

How configurable do all of these nobs even need to be? I'm curious what the use cases are.

RE: .cobalt/

I appreciate trying to get the cruft out of the way but I personally lean against changing the layout in this way. I feel that it isn't so unimportant that they should be put into hidden folders though, out of sight of the user, common search tools, etc. For example, someone might want to search the site for content they see on the page and they should be able to find it in the template/layout without resorting to guessing its in a hidden folder and enabling searching of them. See also my next comment

@johannhof any thoughts on this?

RE: "posts" subfolder

My plan was to actually change the default to _posts and require all collections (besides the root collection) to have a "_" prefix.

My motivation is two part

1. Generalize the posts collection into supporting many user-defined collections
2. Keep everything fast by having something quick to key off of for detecting when the root collection should stop walking the directory hierarchy.

This would be done by leveraging cobalt's ignoring of "_" prefix files and folders rather than getting rid of it. Jekyll set a precedence for this and I think it is a reasonable one to keep with.

[sudoforge]

There is also the awkwardness of the destination being included in the source, requiring special work for cobalt to ignore it. As an aside, in thinking about what are the selling points of cobalt.rs, I figure performance should be one and am looking for how we can design for performance where it doesn't compromise the workflow for the user (that last part being very subjective).

How configurable do all of these nobs even need to be? I'm curious what the use cases are.

Agreed - as far as the ability to configure goes, I think one of the goals of sane, logical defaults is that most users probably won't want/need to configure any of these paths. That said, some people may want to organize their app in a different way -- I think it would be pretty straightforward to have some baked-in default values (which will be needed to populate the initial .yml and file tree when the project is first created anyway), but prefer values in the configuration file (if it exists). Most tools and applications I've used do something similar - like git (and its .git/ directory, .gitconfig file), for instance.

I appreciate trying to get the cruft out of the way but I personally lean against changing the layout in this way. I feel that it isn't so unimportant that they should be put into hidden folders though, out of sight of the user, common search tools, etc. For example, someone might want to search the site for content they see on the page and they should be able to find it in the template/layout without resorting to guessing its in a hidden folder and enabling searching of them. See also my next comment

I can understand your position, and can agree that the potential for the leading dot to hide the skeleton files and templates (and any other files that may be added in the future) might not be ideal (although I'd argue that developers are used to doing config-type-things inside of directories with preceding dots). That said, do you think those subfolders should be exposed at the root level themselves, or simply contained in a "public" directory, like lib/ or cobalt/? I'm a big fan of logical grouping, and to me, skel/ and templates/ are both things that should be thrown into the "stuff used to build your site" folder -- if we expose those, then templates/ would logically (going off of my earlier comment's example) live inside src/, which brings us back to square one in terms of requiring special work to use those files, but not build them.

My plan was to actually change the default to _posts and require all collections (besides the root collection) to have a _ prefix.

I am highly against this change. In terms of a static site, "posts" are really just pages themselves (unlike a dynamic site, where a post may simply be database content pulled into a page). I don't think we should really care how the user organizes their app - what if I want to store my "posts" in a stories/ directory? What if I'm using cobalt to build a static site that doesn't have posts at all? It is for the ease of use and customization that I would suggest that the builder simply builds every .md and .liquid file found in the source directory (in the above example, this is src/), keeping whatever folder hierarchy the user has within it -- which is, the way I understand it, what it is currently doing.

1. Generalize the posts collection into supporting many user-defined collections
2. Keep everything fast by having something quick to key off of for detecting when the root collection should stop walking the directory hierarchy.

Unless I'm misunderstanding you, I think we can do this by simply not caring about "posts", and instead just building everything inside of the source directory. The source file tree defines the "collections"; the user structures their site however they want simply by creating files and folders where they want them to exist.

[epage]

Agreed - as far as the ability to configure goes, I think one of the goals of sane, logical defaults is that most users probably won't want/need to configure any of these paths. That said, some people may want to organize their app in a different way -- I think it would be pretty straightforward to have some baked-in default values (which will be needed to populate the initial .yml and file tree when the project is first created anyway), but prefer values in the configuration file (if it exists). Most tools and applications I've used do something similar - like git (and its .git/ directory, .gitconfig file), for instance.

The challenge is deciding how much to make configurable and in what ways. Should "cobalt/" be configurable? All the "skel", "template", etc folders? It helps to know what problems people are trying to solve to figure out how to structure things.

I can understand your position, and can agree that the potential for the leading dot to hide the skeleton files and templates (and any other files that may be added in the future) might not be ideal (although I'd argue that developers are used to doing config-type-things inside of directories with preceding dots). That said, do you think those subfolders should be exposed at the root level themselves, or simply contained in a "public" directory, like lib/ or cobalt/? I'm a big fan of logical grouping, and to me, skel/ and templates/ are both things that should be thrown into the "stuff used to build your site" folder -- if we expose those, then templates/ would logically (going off of my earlier comment's example) live inside src/, which brings us back to square one in terms of requiring special work to use those files, but not build them.

I do like the idea of centralizing all of the non-content folders separate from content. It cleans things up and we avoid having to ignore these files when walking the content.

[sudoforge]

Should "cobalt/" be configurable? All the "skel", "template", etc folders? It helps to know what problems people are trying to solve to figure out how to structure things.

Ah, I see what you're getting at. In my opinion, no, these paths shouldn't be configurable, as they are used by the builder. We can introduce structure and organization in the form of the cobalt/ directory, or whatever we end up calling it. Since the user never has a reason to want to build these files, I don't think it makes sense to make the subdirectory paths contained within the "build source" directory available in the configuration. It's kind of like git hooks, in a sense -- git doesn't care if you're using hooks or not, but if you want to, you need to put them in .git/hooks/, because that's the directory git is looking at. Similarly, users can opt in to modifying the templates or even adding their own, and modifying skeleton files, or anything else we plop in there simply by navigating to and editing, adding or deleting files.

[epage]

What should be our draft vs published semantics be? Is there value in a separate "_drafts" folder or should we look into other means like setting a "isDraft" property in the front matter?

I started asking this question when considering a cobalt publish command in #208 but isn't coupled to it.

Different ideas on this

• Switching the flag from isDraft to isPublished to make items unpublished by default
• When we add per-collection front-matter defaults, we could have isDraft / isPublished defaulted to different items
• Even weirder ideas like for "posts" to assume its a draft if it doesn't have a date.

[epage]

See also #301 for a more specific case of changing the structure.

[epage]

epage:

My plan was to actually change the default to posts and require all collections (besides the root collection) to have a "" prefix.

I realized why this isn't the case. There are two approaches to structuring a site

• Copy structure as-is
  • Do not prefix the folder with _, causing assets to be copied over
  • use the path relative to the root in your permalink
• Metadata-generated structure
  • The source hierarchy doesn't matter, so make the folder _ leading to draw attention to that.

I think these are both valid cases to support. The question is what is the best way to get all of this right. For example, if a path has a leading _ in one of the components, does that mean we don't offer it for variable substitution?

[epage]

btw #199 and #311 are the core of the changes to the folders including defaults and what is configurable.

As a general rule, one of the things I've started considering is how hard it will be to migrate existing clients when making these breaking changes. That isn't my only reason though. If something is important enough, we'll change it. It just impacts smaller preferences.

RE: layouts -> templates

I'm leaning against this. I feel like layout is more descriptive and template would be better suited for describing the templating done (ie Liquid), see #331.

RE Centralizing the non-content folders.

I'm leaning against this. It is nice to collect all of the non-content folders but adding depth makes things more annoying to edit.

RE . instead of _ to hide these

Most git . folders / files are either implementation details you shouldn't touch (.git) or are support files indirectly related to your work (.gitignore). Neither of these describe the non-content folders in cobalt. They are just as much a part of the site as the documents and assets.

The reason we are hiding these non-content folders is

1. To communicate to the user that we'll not look in them for assets
2. To make it easier for cobalt to not accidentally process them.

While I am going to go ahead and close this, I won't say its the final word. If someone wants to contend any of these points, you are welcome to do so. I am trying to get the major breaking changes out of the way soon (see milestones), so time is relatively short (weeks ideally).