WIP: Convert git-scm.com to use GitHub Pages #942
Comments
|
@spraints Cool! I will be glad to help though I don't know about rails but I do know about Jekyll. If you open up the issues on your repo, it would be awesome. |
|
@maxlazio because Jekyll is better than Middleman :P Faster to compile, develop with, and generally more people know it. Plus it's the main SSG for GitHub Pages. |
This was referenced Feb 7, 2017
|
@maxlazio because of GitHub Pages. It's what I use for static sites, and seemed like a good fit here. @pranitbauva1997 why does https://github.com/spraints/git-scm.com need issues? Pull requests are available, just open a pull request with spraints:gh-pages as the base branch. |
|
@spraints Thanks for your answer, makes sense. I'll send my PR's to the fork and we can coordinate in issues here when necessary. |
|
I like this idea very much. I think its a good idea. But if this becomes a reality, doesn't it make more sense to do a fluent migration? Like having a small PR with adds Jekyll and than i/others can start to migrate it step by step? |
@Sicaine I would like to wait until the site is completely migrated before cutting this repo over to jekyll. The migration can go step-by-step in my fork. For now, I've been manually verifying that the migrated pages look the same on https://git-scm.com/ and http://pages-test-git-scm.pickardayune.com/. I have a small linter script, too. |
|
@spraints i just think that it is harder to maintain two repos up to date. But i also don't know how long it will take. |
This was referenced Feb 9, 2017
|
Had some time so I've sent a couple of PR's. |
|
To be honest - and I don't mean to demean the effort - I don't think this migration is worth it. As I suggested in the email chain, I think it makes more sense to simply generate static assets from the Rails codebase and dump them to a separate GH Pages repo for hosting. The main reason for that is simply because there's already a large body of static assets that need to be generated independently (the man pages and Pro Git) which you would somehow need to add to a Jekyll-based solution. On top of that, although the site is mostly static, there are some dynamic back-end components that need to get fleshed out first, the biggest one being the ElasticSearch layer. |
|
@sxlijin Yes, I think the shortest way to get to a static site is to crawl the generated Rails site and dump it into a static repo for hosting (and in a sense, that's really what a CDN is doing; it's just a big cache). I'm not sure it's a good idea for long-term maintainability, though. Going through Rails makes things a lot more complicated, mostly because the contents of this repository only tell half the story. Most of the actual site content is imported into the database, and its freshness is tracked in a totally separate way (and mostly a primitive one; updating the import code requires manually invalidating the database entries to pull in new versions of the manpages or book content). So if you imagine that the conversion process is to occasionally run: That is heavily dependent on what happens to be in your database at the time you generate. We can work around that, but I think the end result will be simpler if we can just import directly to the filesystem and skip the round-trip through the database. And then we can also perhaps leverage filesystem-aware tools like In a sense, the bits of conversion that have happened so far aren't really the interesting part. There's some tedious work, but it's mostly just converting the templates from one form to another. The heavy lifting, I think, will be the conversion of the manpage and book importers. I dunno. Maybe I am underestimating what value the database is providing to the display code. |
|
I think the page itself is a great example of a good static website. Independend on how often it is regenerated. A nightly build isn't hard and when done properly the heroku issue is gone. But independed from any decision: As long as nothing in particular is decided, its hard to tackle any issues or improvements. I don't wannt to start another approach while this one is going |
The other hard part, I think, is deciding what to do with search. That's the actual dynamic part of the app. I don't know what kinds of 3rd-party solutions we could use there. Obviously linking to I'm pretty unfamiliar with this space in general. That's one of the reasons I was soliciting opinions on the list. :) |
|
It is free and easy to integrate google custom search for a website
|
|
@peff we can dynamically generate the site if we host it with GitLab Pages, which would auomate the currently-manual update process (See this blog post for example). If you don't want to do this that's fine too :P Also we can use Algolia DocSearch for free search, only problem might be that we'd have an Algolia logo in the search results. |
|
To be clear: I'm a fan of Jekyll. I've used it before and I find it, in combination with GH pages, to be absolutely fantastic for its purpose. Pending a resolution to the search question, though, I would be hesitant to move forward with it. (The man pages and book matter too, but since those are simply static assets, there are definitely ways around that.) One friend I spoke to suggested using hosted ElasticSearch if we really want to keep it (which would be 40$/mo from a cursory search) but my honest opinion is that ES is absolutely bonkers overkill for searching the site, especially since as it stands, search doesn't even search page contents, just page titles. And to be frank, that seems like something that's very easily migrated to just one big JS file. |
|
JFTR, I am discussing with another guy who is ready to translate the site to another language. Does migrating to Jekyll would allow providing translated versions of the site? |
|
There's no clear obvious solution (I'm having trouble building the That being said, Jekyll allows you to have data files (stored in |
|
I agree that search is a big wildcard on the static-site transition. Moving the first few individual pages over has been a good validation that the end result looks good (both rendered and in code). But the next step is probably validating those other assumptions (search, imports, etc). I had always assumed we would move to something like Google's site search, but it's possible the "one big JS file" approach might be even simpler. I hadn't given i18n much thought, as the site is not currently translated. But it does seem like an obvious future direction to go in. I'll ping GH pages folks and see what their thoughts are on the state of the art. |
|
I think they're just discontinuing the on-premise Enterprise product. The thing we would use is probably https://cse.google.com/cse/. I hear they are also getting rid of the paid version of that, but I'm not sure we would have used it anyway. That leaves the "ad-supported" search. Which I guess is how normal Google works, but I'm not sure how ugly or intrusive it is on the site. |
|
searching using algolia seems to be a solution worth exploring if we ever go into jekyll: https://blog.frankel.ch/search-static-website/ |
|
👋 I got interested in this project ever since it was announced that Heroku ended its free tier that we use in https://git-scm.com/. We currently have a sponsor, but it would make most sense to spend that money more wisely and making https://git-scm.com/ a nice show case for how powerful GitHub Pages are. Thank you @spraints for starting your incredible work on this, it gave me quite the head start! This is where I'm at right now:
Note: This is just a start. There is still quite a bit to be done.
Plenty of things still to do 😉 |
|
Just a few thoughts, as somebody who has thought about this off and on in the intervening 5 years:
Just my two cents, of course. I haven't looked carefully at the problem in a while. |
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Jun 11, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Jun 11, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Jul 14, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Jul 21, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Jul 21, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: #942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Aug 6, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: git#942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Aug 6, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Aug 6, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Aug 6, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Aug 19, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: #942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Aug 19, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 4, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: git#942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 4, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 7, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 7, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: git#942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 10, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 11, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: git#942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git-scm.com
that referenced
this issue
Sep 11, 2024
Originally, the idea was to use Jekyll instead, see git#942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 11, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
|
@spraints would you agree that we can close this here ticket in favor of 1804? |
dscho
added a commit
that referenced
this issue
Sep 11, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 11, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 11, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 20, 2024
We are about to embark in a major undertaking, an epic undertaking even, seeing as it started in February 2017 with the first attempt to migrate the site to Jekyll: #942. The original motivation was to simplify the setup of the site: a static site is a lot easier to manage and to develop than a Rails App. But in October 2022, a new reason entered the fray: Heroku stopped their free tier and ever since https://git-scm.com/ has required sponsorship whose funding could be put to better use elsewhere. Hence this effort was picked back up almost precisely a year ago by Victoria Dye and myself, and we got really far, so far that pretty much everything worked (or was at least sketched out and proven to work). The major problem was that Jekyll required 20 minutes (!!!) just to render the site. Therefore we pivoted to using Hugo instead, which turned out to be a smart choice: It takes about half a minute to render the entire site. This topic branch starts the migration from the Rails App to a Hugo-generated static site. The main themes of the subsequent migration are: - We will move the original Rails App files that contain Rails code mixed into HTML to `content/`, where the files defining the pages live in the Hugo world, then modify them to drop the Rails code and replace it with Hugo constructs. More often than not, we separate the commits that move the files from the commits that adjust the contents, to help Git realize that there has been a move (as opposed to a delete/add). This allows for noticing upstream changes that need to be reflected in moved & modified files when rebasing to upstream. - In Hugo setups, the files live in the following locations: - `hugo.yml` This is the central configuration file that tells Hugo how to render the site. - `content/` This defines the content of the pages that are served. Only a subset of Hugo's functionality is available here (the idea is to leave the complicated stuff to the layout used to render the pages). Most, but not all, of the files living in this directory tree are HTML files that are generated (and then committed) using external repositories, e.g. the ProGit book and its translations. - `layouts/` This is where the "boiler plate" is defined that ties the site together, i.e. the header, the footer and the sidebar as well as the main scaffolding in which the pages' content is to be rendered. This is the location where most of Hugo's functionality is available and complex stuff can happen such as looping or accessing site parameters. - `layouts/partials/` This directory contains recurring templates, i.e. reusable partial layouts that are used to structure the elements of the site. This includes the side bar, how videos are rendered, etc. - `layouts/shortcodes/` This directory contains so-called "shortcodes", i.e. reusable elements similar to partial layouts. The major difference is that shortcodes can be used within `content/` while partial layouts can only be used from within `layouts/`. See https://gohugo.io/content-management/shortcodes/ for more information on this topic. - `static/` These files are not processed by Hugo, but copied as-are. Good for images, for example. - `assets/` These files are processed in specific ways. That is where the SASS-based style sheets live, for example. - `data/` These files define metadata that can be used in Hugo's functions. For example, it will contain the list of documentation categories that are rendered in various ways. - In contrast to most Hugo-managed sites, we will refrain from using a Hugo theme, and instead stick with the existing style sheets. Likewise, we refrain from using Markdown at all: The existing site did not use it, therefore it makes little sense to start using it now. - In addition to Hugo's directories, we also have these: - `script/` This directory contains scripts to perform recurring tasks such as rendering Git's manual pages into HTML that are then stored inside `contents/docs/`. For historical reasons, these are Ruby scripts for the most part, as it is easier to follow the development when that functionality is extracted from the current Rails App and turned into Ruby scripts that can be run stand-alone. - `.github/workflows/` and `.github/actions/` The latter directory contains a file that defines a custom GitHub Action that accommodates for the lack of Hugo support in GitHub Pages: By default, only Jekyll pages are supported out of the box, but Hugo sites require a custom GitHub workflow to deploy the site. The former directory contains files that define GitHub workflows that are typically run on a schedule, updating the various parts that are generated from external sources: the Git version, the ProGit Book, manual pages, etc. These workflows essentially keep the rendered HTML files in `content/` up to date with the respective external repositories. - `_generated-asciidoc/` This directory serves as a cache of "expanded AsciiDoc": many of Git's manual pages include content from other files, and therefore it is non-trivial to determine whether or not a manual page has changed and needs to be re-rendered (essentially, the only way is to expand them by inlining the `include`d files). Caching this content speeds up updating the manual pages drastically. - Most of the core logic lives in `layouts/`. Hugo discerns between logic that is allowed in `layouts/` and logic that is allowed in `content/`; The latter can only access so-called "shortcodes" https://gohugo.io/content-management/shortcodes/. These shortcodes are free to use the entire set of Hugo's functionality. tl;dr whenever we need to do something complicated that is confined to only a few pages, we have to implement it in `layouts/shortcodes/` and insert the corresponding `{{< shortcode-name >}}` in the page itself. Whenever we need to something complicated that is used in more places, it is implemented elsewhere in `layouts/`. - Some of the logic that cannot be performed statically (such as telling the user how long ago the latest macOS installer was released, or adjusting the Windows downloads to reflect the CPU architecture indicated by the current user agent) are implemented using Javascript instead. - The site search needs to move to the client side, as there is no longer a server that can perform that functionality. Luckily, Pagefind (https://pagefind.app/) matured in the meantime, a very performant client-side search solution implemented in Javascript that relies on a search index that is generated at build time and that is served incrementally, as needed, via static files. This is what we use, then. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
that referenced
this issue
Sep 20, 2024
Originally, the idea was to use Jekyll instead, see #942. However, I aborted that migration when it turned out that Jekyll required 20 minutes to process the files while Hugo spent less than half a minute on them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
|
Thank you for all your hard work, @spraints! For the record, we already tried to switch over this past Friday, but due to deployment issues as well as ugly problems due to mismatching cached assets, we had to roll back. I am optimistic, though, that my most recent changes address all o' that. |
|
@spraints the Hugo/Pagefind-backed site is live now! (You can verify it e.g. by running a search for a specific language, e.g. https://git-scm.com/search/results?search=commit&language=es). |
|
🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 Nice! @dscho You deserve a ton of credit too for making sure it got finished! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Re @peff's recent ML message, I started playing around with converting this site to be a Jekyll site, so it can be hosted on GitHub Pages instead of heroku. https://github.com/spraints/git-scm.com is the new code, and http://pages-test-git-scm.pickardayune.com/ is the rendered site. So far, I've converted the home page and the "about" pages. My goal is for the pages site to be able to handle all of the same URLs that the current site knows about. Also, I'm not a designer, so I'm also shooting to make the site look exactly identical.
The home page was a fairly mechanical conversion. In the rails app, the "about" page is a single page that rewrites its URLs; I split it up so the links are normal links.
I'll continue to poke at it in my spare time. I'd also accept pull requests 😻 to my
gh-pagesbranch. It should be pretty easy to figure out what needs to be done: find a broken link on http://pages-test-git-scm.pickardayune.com/, and copy content from the rails app in https://github.com/git/git-scm.com to the right place in the jekyll app. Rails helpers need to be changed into flat HTML or liquid tags. I hear that there are man-pages in the current site's database, which may take some more effort to convert.The text was updated successfully, but these errors were encountered: