Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call me collections #2199

Merged
merged 30 commits into from Apr 15, 2014

Conversation

Projects
None yet
8 participants
@parkr
Copy link
Member

parkr commented Apr 2, 2014

  • Use Collections for data feature
  • Allow us to read in collections
  • Render collections
  • TomDoc the sh*t outta this code
  • Write docs
  • Expose to Liquid via site.*

/cc @gjtorikian

Fixes #1941.

And I have a configuration file with:
| key | value |
| collections | ['methods'] |
| render | \n methods: /methods/:subdir/:title:extname |

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

This compiles to:

collections: ["methods"]
render:
  methods: /methods/:subdir/:title:extname

Specifying the output URL only in one key rather than trying to clump things and over-complicate collections key.

Thoughts on this, @gjtorikian & @benbalter?

This comment has been minimized.

@gjtorikian

gjtorikian Apr 2, 2014

Member

It makes sense. The : notation leaves a bit to be desired. I think what bothers me is seeing it in such close succession: :title:extname. Is this the only possibility, or can another syntax be used?

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

@gjtorikian Other option is to do string interpolation:

render:
  methods: "/methods/#{subdir}/#{title}#{extname}"

Would just use ERB to inject the variable values and boom done.

Downside: inconsistent.

This comment has been minimized.

@benbalter

benbalter Apr 2, 2014

Contributor

Really like this. Two questions:

  1. What's subdir?
  2. Don't :'s in values bork YAML sans quotes?

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member
  1. :subdir is the subdirectory that contains the file, i.e. the folders b/w the collection folder and the file. E.g. jekyll is the subdir for _methods/jekyll/configuration.md.

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

shouldn't that be :path?

:path would include the filename.

And when would I want any configuration other than :path/:file:extension?

When I want to prefix all the files with something.

This comment has been minimized.

@benbalter

benbalter Apr 2, 2014

Contributor

When I want to prefix all the files with something.

Do you have an example that would be >= 20% of the use cases? Feels like an edge case to me that adds a lot of complexity and could be resolved by simply using a different naming pattern for the file.

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

So in the case of _methods/site/generate.md, what would you expect the output path to be?

This comment has been minimized.

@benbalter

benbalter Apr 2, 2014

Contributor

So options one, it's always /:collection_name/:path, unless you specify a permlink. Option two, you specify a permalink structure of /:collection/:path and get /methods/site/generator?

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

Yeah those are the options. The question is whether we should allow the user to specify. Maybe that's a 2.1.0 optimization. PUNT. 🏈

@parkr parkr added this to the 2.0 milestone Apr 2, 2014

@parkr parkr added the Feature label Apr 2, 2014

{
"content" => output,
"page" => document.to_liquid,
"layout" => layout.data

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

@penibelst What if I add this for layouts? Re: #2070

This comment has been minimized.

@doktorbro

doktorbro Apr 3, 2014

Member

@parkr Sorry, I can’t judge if it solve the issue.


Put some things in a folder and add the folder to your config. It's simple...

Why did we write this feature? What is it useful for?

This comment has been minimized.

@parkr

parkr Apr 2, 2014

Author Member

@benbalter @gjtorikian HALP i have no idea what to write in the docs besides what is below.

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Not everything is a post or a page. Maybe you want to document the various methods in your open source project, members of a team, or talks at a conference. Collections allow you to define a new type of document that behave like Pages or Posts do normally, but also have their own unique properties and namespace.


## Using Collections

### Step 1: Tell Jekyll to read in your collection

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Add the following to your site's _config.yml file, replacing my_collection with the name of your collection:


### Step 2: Add your content

Create a corresponding folder (e.g. `<source>/_my_collection`) and add documents.

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Note, the folder must be named identical to the collection you defined in you config.yml file, with the addition of the preceding _ character.

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

Added in a10755a

### Step 2: Add your content

Create a corresponding folder (e.g. `<source>/_my_collection`) and add documents.
YAML front-matter is read in as data if it exists, if not, then everything is just

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

may want to clarify it gets read in as post._____ not site.data.______.


### Step 3: Optionally render your collection's documents into independent files

If you'd like your files rendered, add it to your config:

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

If you'd like Jekyll to create a public-facing version of each document in your collection...

end

def relative_path
Pathname.new(path).relative_path_from(Pathname.new(site.source)).to_s

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

We sanitize the collection name? e.g., I can't make a collection ../../etc/password?

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

Was thinking about this yesterday. Was thinking collection names should only be allowed to contain alphanumeric characters, underscores and hyphens.

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

Just added it in 4a136c5

screen shot 2014-04-03 at 2 11 26 pm

This comment has been minimized.

@benbalter

benbalter Apr 4, 2014

Contributor

collection names should only be allowed to contain alphanumeric characters, underscores and hyphens.

👍

data_collection.read
data_collection.docs.each do |doc|
key = sanitize_filename(doc.basename(".*"))
self.data[key] = doc.data

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Why hardcode this? Shouldn't this happen automatically for any collection?

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

I tried doing that initially but data works a bit differently. Not sure the best way to handle this yet.

This comment has been minimized.

@parkr

parkr Apr 6, 2014

Author Member

The problem is that the files are accessed via site.data.filename_without_extname which isn't the current site.posts/site.pages paradigm we went with here.


key = sanitize_filename(File.basename(entry, '.*'))
self.data[key] = SafeYAML.load_file(path)
data_collection = Jekyll::Collection.new(self, "data")

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Why not just site.collections.merge! ["data"]?

@benbalter

This comment has been minimized.

Copy link
Contributor

benbalter commented Apr 3, 2014

The vision is to eventually 🐶food this for Pages and Posts, right? 😄

@benbalter

This comment has been minimized.

Copy link
Contributor

benbalter commented Apr 3, 2014

May want to make notes in the docs that this is an experimental feature and that the API may likely change until the feature stabilizes. Can help manage expectations and give us a bit more flexibility around SemVer.

@benbalter benbalter referenced this pull request Apr 3, 2014

Closed

WIP: Implement Arbitrary Collection Support #2028

2 of 9 tasks complete
@parkr

This comment has been minimized.

Copy link
Member Author

parkr commented Apr 3, 2014

@benbalter Thank you times a million for reading through this and commenting. So freakin helpful.

The vision is to eventually 🐶food this for Pages and Posts, right?

ja, but I want to get an initial version out there. There are a larrrrge # of further questions in order to turn pages and posts into collections (like using the Page and Post classes) so I'd rather do that in a different PR.

{
"label" => label,
"docs" => docs
}

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

@benbalter Do you think we should just send back the docs array instead of the label, too? It then behaves like site.posts:

{% for datafile in site.collections.data %}
  {{ datafile.foo }} {{ datafile.bar }}
{% endfor %}

This comment has been minimized.

@benbalter

benbalter Apr 3, 2014

Contributor

Let's not make things harder down the line, or build out a new design pattern.

I'd expose collections as a first class citizen like we do posts or pages, so a documents collection should live as site.documents and I should iterate through it just like I would site.posts or site.pages.

This comment has been minimized.

@parkr

parkr Apr 3, 2014

Author Member

"a first class citizen"

so a documents collection should live as site.documents

I'm really weary of this in terms of configuration collisions or future features. What's so bad about namespacing them in site.collections?

This comment has been minimized.

@benbalter

benbalter Apr 4, 2014

Contributor

What's so bad about namespacing them in site.collections?

Because pages, posts, and data would be first-class citizens in Jekyll's eyes, but my collections would be a second class citizen (Not to mention, would make dogfooding 999x harder).

User is in control of both site config vars and collection names. Assuming that a collection name is plural, the potential collisions you're looking at are plugins, layouts, gems, which I'd argue, collections could just overwrite, as those three in particular are rarely access in userspace.

This comment has been minimized.

@parkr

parkr Apr 4, 2014

Author Member

User is in control of both site config vars and collection names

Alright... this puts my mind at ease. My greatest worry about this is obviousness – will the user intuit access to these collections? If not, how can we get them as close as possible to just knowing that a given collection can be accessed in this way with this variable reference?

This comment has been minimized.

@benbalter

benbalter Apr 4, 2014

Contributor

Answering your question, and another advantage is education is easier. "Collections are just like posts or pages, but you choose what they're called". Behavior is identical to existing behavior, no new site.collections.foo design/interaction pattern to learn.


# An inspect string.
#
# Returns the inspecr string

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Minor: spelling of inspect


# Create a new Document.
#
# shit - the Jekyll::Site instance to which this Document belongs

This comment has been minimized.

@baweaver
# Returns the permalink or nil if no permalink was set in the data.
def permalink
return nil if data.nil? || data['permalink'].nil?
data['permalink']

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

If you make this

data && data['permalink']

...it eliminates the need for the return nil up there

@url ||= URL.new({
:template => url_template,
:placeholders => url_placeholders,
:permalink => permalink

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Minor, but json syntax for consistencies sakes with lines 104 - 106

#
# Return the file read options hash.
def merged_file_read_opts(opts)
(site ? site.file_read_opts : {}).merge(opts)

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Might get a bit more efficiency out of it like:

site ? site.file_read_opts.merge(opts) : opts
"page" => document.to_liquid
}, site.site_payload)

info = {

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Minor: extra spaces

raise e
end
end
output

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Could be a bit cleaner like:

content.map { |raw_output|
  converters.reduce(raw_output) { |output, converter|
    begin
      converter.convert output
    rescue => e
      Jekyll.logger.error "Conversion error:", "#{converter.class} encountered an error converting '#{document.relative_path}'."
      raise e
    end
  }
}
docs = docs.merge(coll.docs)
end
end
docs

This comment has been minimized.

@baweaver

baweaver Apr 6, 2014

Reduce can help shorten quite a few methods.

collections.reduce(Set.new) { |docs, (label, coll)|
  to_render.include?(label) ? docs.merge(coll.docs) : docs
}
<div class="note warning">
<h5>Collections support is unstable and may change</h5>
<p>
This is an experimental feature and that the API may likely change until the feature stabilizes.

This comment has been minimized.

@gjtorikian

gjtorikian Apr 7, 2014

Member

nix that that

This comment has been minimized.

@parkr

parkr Apr 14, 2014

Author Member

suggested by @benbalter


Put some things in a folder and add the folder to your config. It's simple...

Not everything is a post or a page. Maybe you want to document the various methods in your open source project, members of a team, or talks at a conference. Collections allow you to define a new type of document that behave like Pages or Posts do normally, but also have their own unique properties and namespace.

This comment has been minimized.

@gjtorikian

gjtorikian Apr 7, 2014

Member

Collections allow you to define a new type of document that behave like Pages or Posts do normally, but also have their own unique properties and namespace.

It took me a while to parse this sentence. What about minimizing it, such as:

Collections allow you to define documents that behave like Pages or Posts, but also have their own unique properties and namespace. You can iterate over your site's unique collections just like you can with Pages and Posts.

Kind of lame on the last sentence, but I wanted to draw a closer parallel between P & P + C.

### Step 2: Add your content

Create a corresponding folder (e.g. `<source>/_my_collection`) and add documents.
YAML front-matter is read in as data if it exists, if not, then everything is just stuck in the Document's `content` attribute.

This comment has been minimized.

@gjtorikian

gjtorikian Apr 7, 2014

Member

YAML front-matter is read in as data if it exists; if there's no front-matter, then everything is just stored as the Document's content attribute.

How can the metadata be accessed? collection.metadata_var ?

This comment has been minimized.

@parkr

parkr Apr 8, 2014

Author Member

It's addressed below. I'll be more explicit about pointing people that way :)

</td>
<td>
<p>
An array of Documents contained in this collection.

This comment has been minimized.

@gjtorikian

gjtorikian Apr 7, 2014

Member

Meta question: so to iterate over the collection, I need to do site. collections.docs ? Any reason it can't be site.<my_collection_name>?

parkr added a commit that referenced this pull request Apr 15, 2014

@parkr parkr merged commit cb4a7a5 into master Apr 15, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details

@parkr parkr deleted the collection-plate branch Apr 15, 2014

parkr added a commit that referenced this pull request Apr 15, 2014

@benbalter

This comment has been minimized.

Copy link
Contributor

benbalter commented Apr 15, 2014

YESSSSSSSSSSS

@jekyll jekyll locked and limited conversation to collaborators Feb 27, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.