Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make possible to weblorg-copy-static copy other files than the static directory #32

Open
semente opened this issue Feb 21, 2021 · 11 comments
Labels
design Issues that still need some design discussion before being implemented enhancement New feature or request

Comments

@semente
Copy link
Contributor

semente commented Feb 21, 2021

Would be interesting if weblorg-copy-static could copy things outside a predefined static directory. For instance, I use this in org-publish:

("assets"
  :base-directory "."
  :base-extension "jpg\\|png\\|txt\\|sh\\|el"
  :exclude '("^_" "^[.#]" "~$")
  :recursive t
  :publishing-directory ".build"
  :publishing-function org-publish-attachment)

With this I can keep post assets in the same directory of the posts or in any other directory of my choice. Once again, as mentioned in other issues I have open, I want to be able to easily browse my project files using Emacs in the same way as in a web browser.

With weblorg I currently copy these files using rsync but weblorg-copy-static could support something like this:

(weblorg-copy-static
 :base-dir "."   ; other than theme/static
 :input-pattern '("**/.jpg" "**/*.png" "**/*.txt" "**/*.sh" "**/*.el")
 :output "output/{{ file }}"
 :url "/{{ file }}")

Besides the custom input-pattern and base-dir, note the multiple input values using a list.

@JensAc
Copy link

JensAc commented Feb 25, 2021

Hey,
I want to add something on this. Would it make sense to have a directory per per post. Let's say I have code blocks, which export figures and I have a lot of posts doing this. Then, I maybe do not want to have all the images in the same directory, as this would be really confusing, when browsing the directory. Perhaps, I even do not want to name the images differently.
Would that contradict the overall concept of weblorg?

@mrmechko
Copy link
Contributor

Pull requests adds :data-dir option to weblorg-copy-static to allow copying individual static directories over into the final weblorg site.

The problem I was having was that simply providing :base-dir resulted into a translation to a theme directory within :base-dir. However, that is a very useful feature, so instead of messing with the theme translation, I just added the :data-dir option. If you have a :data-dir, that particular static route will copy files in :data-dir. You can still use :input-filter etc as you see fit.

(weblorg-copy-static
 :name "cs141-static"
 :data-dir "~/docs/141/static"
 :output "output/cs141/static/{{ file }}"
 :url "/static/{{ file }}")

@clarete clarete added the enhancement New feature or request label Mar 1, 2021
@clarete
Copy link
Collaborator

clarete commented Mar 1, 2021

I'm interested in this use-case. I'm under the impression that we're missing a small piece to make it all work, but I wanted to see if my assumptions are correct.

If I am getting this right, you want something that 1. goes through a directory recursively, 2. finds files matching a list of patterns, 3. copy them to a destination directory that follows the same directory structure. If that is correct, I think the weblorg--copy-static that we currently have is short of an option like :recursive t, just like org-publish does. @semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))? do you think we could keep :input-pattern the way it is if we use it with regexp-opt?

And, more objectively, would :recursive t solve this issue for you guys?

@mrmechko
Copy link
Contributor

mrmechko commented Mar 1, 2021

I particularly want to be able to move static files that are not theme files (eg linked images, or slides). :data-dir specifies a file that overrides the location finding function weblorg--path. Is there better way to do that?

@semente
Copy link
Contributor Author

semente commented Mar 1, 2021

@semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))?

hi @clarete! yea, in combination with :recursive t it would work just fine. My thought was since :input-pattern already support the globstar ** it would make more sense than adding another option (:recursive).

So I think we already have an recursive t if we can use **/*. I would just filter in few files based on extensions (or regex).

@clarete
Copy link
Collaborator

clarete commented Mar 2, 2021

@semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))?

hi @clarete! yea, in combination with :recursive t it would work just fine. My thought was since :input-pattern already support the globstar ** it would make more sense than adding another option (:recursive).

So I think we already have an recursive t if we can use **/*. I would just filter in few files based on extensions (or regex).

ohh, that's pretty interesting. I'm going to see if the eshell function that I use to resolve the glob expression can expand recursively as well. That would save us from adding a new parameter, which is more expensive in terms of maintenance/documentation/etc!

@clarete
Copy link
Collaborator

clarete commented Mar 2, 2021

I particularly want to be able to move static files that are not theme files (eg linked images, or slides). :data-dir specifies a file that overrides the location finding function weblorg--path. Is there better way to do that?

after learning about this globstar thing, which I didn't really fully understand, my plan is to try to use just the :input-pattern parameter as a way of finding which files need to be copied. That will take removing the hardcoded mention to the "static" directory there. If the globstar thing doesn't work, we'll probably have to add the :recursive t parameter alongside with the fix for using `:input-pattern' here.

@semente
Copy link
Contributor Author

semente commented Mar 4, 2021

@clarete weblorg use eshell-extended-glob, which has globstar capability. See for instance (eshell-extended-glob "/etc/**/*.conf"), it will match all the /etc/*.conf, /etc/*/*.conf, /etc/*/*/*.conf files, and so on.

@clarete
Copy link
Collaborator

clarete commented Sep 6, 2021

Hi hi hi 👋🏾 this has taken some time to sink in for sure. Really sorry about that and thank you for the patience, but I think I've finally got an idea worth sharing 😄

It really took me trying to setup a few different static websites until I really saw what you guys were saying about wanting to keep "the original filesystem structure" of the source of a website. That is a very sophisticated feature that I didn't really have in mind while designing the first version of weblorg. This is a very good idea and very powerful feature to be included but it might take a bit more of thought and some emacs lisp hacking to get done, so I feel like it's fair to say that it'd be a feature for the 0.3 release. (0.2 is sorta coming 😄).

With all that said, this is what I thought that we could do to achieve what I think we all want here:

(weblorg-route
 ;; Name of the route.  That is what we use to refer to this
 ;; route within other routes via `url_for' for example.
 :name "index"

 ;; Find files ending with all these extensions
 :input-pattern "posts/*.(jpg|png|css|org)"

 ;; Declare a different handler for each
 :input-handle '((:pattern "*.(jpg|png|css)"
                  :handler #'weblorg-handle-static)

                 (:pattern "*.org"
                  :aggregate #'weblorg-input-aggregate-each
                  :handler #'weblorg-handle-orgmode
                  :template "post.html"
                  :output-filename #'weblorg-output-filename-slugified))

 ;; All input files will have the same base directory as the
 ;; output.  That said, you can still customize how the file
 ;; name of each input handle type will look like.
 :output "output/{{ output_file_name }}"
 :url "/")

The above proposal of including the new :input-handle parameter that allows
one single route to have different handlers. That's how we'd split the pipeline
for different inputs. The splitting of the list of input files with a matching of a
regular expression against the file path of the input that matched :input-pattern.

Also, notice the :output-filename parameter. That is a response to the issue #27,
so you'd be able to configure how the file name of each handler would be created.

Curious to hear what you guys think 😄

/cc @semente @guilhermecomum

@clarete clarete added the design Issues that still need some design discussion before being implemented label Sep 6, 2021
@semente
Copy link
Contributor Author

semente commented Sep 11, 2021

@clarete :D

Yea, I really like this!

Really sorry about that and thank you for the patience, but I think I've finally got an idea worth sharing smile

There is no rush! Don't worry. Thank you

@nanzhong
Copy link
Contributor

I like the described approach!

I use a similar structure in my personal site, with each post/page having its own directory that includes the org file and assets the org file links to. My current approach is to ignore the static assets in weblorg and have an out of band process to copy those over preserving the output structure that weblorg uses. This feels like it could replace that entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design Issues that still need some design discussion before being implemented enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants