From 3fa0ac675baa7e28795cee856455a76d85007171 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Wed, 17 Sep 2025 21:31:19 +0200 Subject: [PATCH 01/13] feat: add self-host instructions Closes #827 Signed-off-by: Gil Desmarais --- src/content/docs/get-involved/index.mdx | 3 +- .../docs/get-involved/self-hosting.mdx | 40 +++++++++++ .../docs/troubleshooting/troubleshooting.mdx | 67 ++++++++++++++++--- .../web-application/how-to/deployment.mdx | 45 +++++++++++-- 4 files changed, 138 insertions(+), 17 deletions(-) create mode 100644 src/content/docs/get-involved/self-hosting.mdx diff --git a/src/content/docs/get-involved/index.mdx b/src/content/docs/get-involved/index.mdx index f56546ea..1e4c82f7 100644 --- a/src/content/docs/get-involved/index.mdx +++ b/src/content/docs/get-involved/index.mdx @@ -8,7 +8,8 @@ sidebar: Engage with the `html2rss` project. Contribute and connect with the community. - [**Project Roadmap**](https://github.com/orgs/html2rss/projects/3/views/1): View current work, plans, and priorities. +- [**Self-Host Your Own Instance**](/get-involved/self-hosting): Take control of your information diet and join the decentralized web movement. - [**Report Bugs & Discuss Features**](/get-involved/issues-and-features): Report bugs or propose features. - [**Join Community Discussions**](/get-involved/discussions): Connect with users and contributors. - [**Contribute to html2rss**](/get-involved/contributing): Contribute code, documentation, or feed configurations. -- [**Sponsoring**](/get-involved/sponsoring) +- [**Sponsoring**](/get-involved/sponsoring). diff --git a/src/content/docs/get-involved/self-hosting.mdx b/src/content/docs/get-involved/self-hosting.mdx new file mode 100644 index 00000000..e95098a9 --- /dev/null +++ b/src/content/docs/get-involved/self-hosting.mdx @@ -0,0 +1,40 @@ +--- +title: "Self-Host Your Own Instance" +description: "Take control of your information diet. Host your own html2rss instance and join the decentralized web movement." +sidebar: + order: 3 +--- + +Turn any website into an RSS feed. Self-host your own instance and take back control of your information diet. + +## Quick Start + +1. Install [Docker](https://docs.docker.com/get-docker/). +2. Create a directory for html2rss: + ```bash + mkdir html2rss-web && cd html2rss-web + ``` +3. Download the configuration: + ```bash + curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/docker-compose.yml + curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/config/feeds.yml + ``` +4. Start the instance: + ```bash + docker compose up -d + ``` +5. Visit [http://localhost:3000](http://localhost:3000) + You should see the html2rss-web interface. + +## Next Steps + +- [Secure your instance](/web-application/how-to/deployment#security) +- [Enable HTTPS](/web-application/how-to/deployment#https) +- [Troubleshoot issues](/troubleshooting/troubleshooting) +- [Join the community](https://github.com/orgs/html2rss/discussions) + +--- + +## License + +[MIT](https://github.com/html2rss/html2rss/blob/main/LICENSE) diff --git a/src/content/docs/troubleshooting/troubleshooting.mdx b/src/content/docs/troubleshooting/troubleshooting.mdx index b8c16832..536e260a 100644 --- a/src/content/docs/troubleshooting/troubleshooting.mdx +++ b/src/content/docs/troubleshooting/troubleshooting.mdx @@ -11,7 +11,9 @@ Your browser's developer tools are essential for troubleshooting. Use them to in - **To open:** Right-click an element on a webpage and select "Inspect" or "Inspect Element." -## Common Issues +--- + +## Common Issues (Ruby Gem / CLI) ### Empty Feeds @@ -21,7 +23,7 @@ If your feed is empty, check the following: - **`items.selector`:** Verify that the `items.selector` matches the elements on the page. - **Website Changes:** Websites change their HTML structure frequently. Your selectors may be outdated. - **JavaScript Content:** If the content is loaded via JavaScript, use the `browserless` strategy instead of `faraday`. -- **Authentication:** Some sites require authentication - check if you need to add headers or use a different strategy. +- **Authentication:** Some sites require authentication — check if you need to add headers or use a different strategy. ### Configuration Errors @@ -54,21 +56,68 @@ If you are getting a "command not found" error, try the following: - **Re-install:** Re-install `html2rss` to ensure it is installed correctly: `gem install html2rss`. - **Check `PATH`:** Ensure that the directory where Ruby gems are installed is in your system's `PATH`. -### Web Application Errors - -For html2rss-web specific issues: +--- -- **`401 Unauthorized`:** Check your `AUTO_SOURCE_USERNAME` and `AUTO_SOURCE_PASSWORD` environment variables. -- **`403 Forbidden`:** The URL is not in the `AUTO_SOURCE_ALLOWED_URLS` list, or the origin is not in `AUTO_SOURCE_ALLOWED_ORIGINS`. -- **`500 Internal Server Error`:** Check the application logs for detailed error information. +## Web Application Issues (html2rss-web) + +### Instance Won’t Start + +- Verify Docker is installed and running: + ```bash + docker --version + ``` +- Check logs for errors: + ```bash + docker compose logs + ``` +- Ensure the port (default: 3000) isn’t already in use: + ```bash + netstat -tulpn | grep :3000 + ``` + +### Can’t Access the Web Interface + +- Confirm your firewall allows traffic on port 3000 (or the port you configured) +- Try accessing via the server’s IP instead of a domain name +- Double-check that containers are running: + ```bash + docker compose ps + ``` + +### Authentication Errors + +- **401 Unauthorized:** Check your `AUTO_SOURCE_USERNAME` and `AUTO_SOURCE_PASSWORD` environment variables. +- **403 Forbidden:** The URL is not in the `AUTO_SOURCE_ALLOWED_URLS` list, or the origin is not in `AUTO_SOURCE_ALLOWED_ORIGINS`. +- **500 Internal Server Error:** Check the application logs for detailed error information. - **Health check failures:** Use the `/health_check.txt` endpoint to identify which specific feed configurations are broken. +### SSL / HTTPS Issues + +- Follow the [nginx reverse proxy guide](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) +- Use [Let’s Encrypt with certbot](https://certbot.eff.org/) to obtain valid certificates +- Verify your reverse proxy forwards requests to the correct internal port + +### Feed Problems + +- Some sites may require JavaScript rendering; ensure the `browserless` service is running +- Check the feed configuration in `feeds.yml` for typos or invalid selectors +- Look for parsing errors in the logs: + ```bash + docker compose logs html2rss-web + ``` + +--- + ## Tips & Tricks - **Mobile Redirects:** Check that the channel URL does not redirect to a mobile page with a different markup structure. - **`curl` and `pup`:** For static sites, use `curl` and `pup` to quickly find selectors: `curl URL | pup`. - **CSS Selectors:** For a comprehensive overview of CSS selectors, see the [W3C documentation](https://www.w3.org/TR/selectors-4/#overview). +--- + ## Still Stuck? -If you are still having issues, please [open an issue on GitHub](https://github.com/html2rss/html2rss/issues). +- Join our [community discussions](https://github.com/orgs/html2rss/discussions) +- [Open an issue on GitHub](https://github.com/html2rss/html2rss/issues) +- Review the [deployment guide](/web-application/how-to/deployment) for production best practices diff --git a/src/content/docs/web-application/how-to/deployment.mdx b/src/content/docs/web-application/how-to/deployment.mdx index c1e556d2..e338238c 100644 --- a/src/content/docs/web-application/how-to/deployment.mdx +++ b/src/content/docs/web-application/how-to/deployment.mdx @@ -3,26 +3,55 @@ title: "Deployment & Production" description: "Deploy html2rss-web to production with Docker. Learn best practices for hosting public instances with security, monitoring, and reliability." --- -html2rss-web is published on Docker Hub, making it easy to deploy with Docker. The `docker-compose.yml` from our [Installation Guide](/web-application/getting-started) provides a solid foundation for both development and production use. +html2rss-web is published on Docker Hub, making it easy to deploy with Docker. The [`docker-compose.yml`](https://github.com/html2rss/html2rss-web/blob/master/docker-compose.yml) from our [Installation Guide](/web-application/getting-started) provides a solid foundation for both development and production use. ## Quick Start -If you've already set up html2rss-web locally, you're ready to deploy it to production. The same Docker configuration works for both environments. +1. Create a directory for your instance: + ```bash + mkdir html2rss-web && cd html2rss-web + ``` +2. Download the configuration: + ```bash + curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/docker-compose.yml + curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/config/feeds.yml + ``` +3. Start the instance: + ```bash + docker compose up -d + ``` +4. Open [http://localhost:3000](http://localhost:3000) to verify it’s running. + +> 📖 Need Docker? Follow the [Docker installation guide](https://docs.docker.com/get-docker/) and [Docker Compose install](https://docs.docker.com/compose/install/). ## Production Best Practices When hosting a **public instance** that others will use, please follow these essential guidelines: -### Security Essentials +### Security -- **Use a reverse proxy** (nginx, Apache, or Cloudflare) to handle SSL termination and rate limiting -- **Enable HTTPS only** - redirect all HTTP traffic to HTTPS +- **Use a reverse proxy** ([nginx](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/), Apache, or Cloudflare) to handle SSL termination and rate limiting +- **Enable HTTPS only** — redirect all HTTP traffic to HTTPS ([certbot guide](https://certbot.eff.org/)) - **Set strong passwords** for health check and auto-source authentication - **Restrict access** to admin endpoints and sensitive configuration +### HTTPS + +Setting up HTTPS is crucial for any public instance. Follow these steps: + +1. **Obtain SSL certificates** using [Let's Encrypt](https://letsencrypt.org/) with [Certbot](https://certbot.eff.org/) +2. **Configure your reverse proxy** to handle SSL termination +3. **Redirect all HTTP traffic** to HTTPS automatically +4. **Test your setup** using tools like [SSL Labs](https://www.ssllabs.com/ssltest/) + +For detailed implementation guides, see: +- [Nginx SSL configuration](https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/) +- [Apache SSL setup](https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html) +- [Cloudflare SSL/TLS settings](https://developers.cloudflare.com/ssl/) + ### Reliability & Monitoring -- **Enable auto-updates** using watchtower or similar tools +- **Enable auto-updates** using [watchtower](https://containrrr.dev/watchtower/) or similar tools - **Monitor the health check endpoint** (`/health_check.txt`) to detect issues early - **Set up logging** to track errors and performance - **Use environment variables** for configuration instead of hardcoded values @@ -31,7 +60,7 @@ When hosting a **public instance** that others will use, please follow these ess - **Configure appropriate resource limits** for your Docker containers - **Use a CDN** for static assets if serving many users -- **Monitor memory usage** - html2rss-web can be memory-intensive with large feeds +- **Monitor memory usage** — html2rss-web can be memory-intensive with large feeds - **Set up caching** for frequently accessed feeds ## Environment Configuration @@ -47,6 +76,8 @@ environment: BASE_URL: https://your-domain.com ``` +Use a [password manager](https://bitwarden.com/password-generator/) to generate strong values. + ## Share Your Instance Once your instance is running smoothly: From 6491a263d64252e19a616a38f0b625a0128f201a Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Wed, 17 Sep 2025 21:48:34 +0200 Subject: [PATCH 02/13] improve deployment knowledge --- .../web-application/how-to/deployment.mdx | 129 ++++++++++++++++-- 1 file changed, 118 insertions(+), 11 deletions(-) diff --git a/src/content/docs/web-application/how-to/deployment.mdx b/src/content/docs/web-application/how-to/deployment.mdx index e338238c..0add7db9 100644 --- a/src/content/docs/web-application/how-to/deployment.mdx +++ b/src/content/docs/web-application/how-to/deployment.mdx @@ -30,11 +30,17 @@ When hosting a **public instance** that others will use, please follow these ess ### Security -- **Use a reverse proxy** ([nginx](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/), Apache, or Cloudflare) to handle SSL termination and rate limiting -- **Enable HTTPS only** — redirect all HTTP traffic to HTTPS ([certbot guide](https://certbot.eff.org/)) +- **Use a reverse proxy** to handle SSL termination and rate limiting +- **Enable HTTPS only** — redirect all HTTP traffic to HTTPS - **Set strong passwords** for health check and auto-source authentication - **Restrict access** to admin endpoints and sensitive configuration +### Modern Reverse Proxy Options + +- **Caddy** — Automatic HTTPS, zero config +- **Traefik** — Auto-discovery, perfect for Docker +- **nginx** — Traditional, requires certbot setup + ### HTTPS Setting up HTTPS is crucial for any public instance. Follow these steps: @@ -45,38 +51,132 @@ Setting up HTTPS is crucial for any public instance. Follow these steps: 4. **Test your setup** using tools like [SSL Labs](https://www.ssllabs.com/ssltest/) For detailed implementation guides, see: + - [Nginx SSL configuration](https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/) - [Apache SSL setup](https://httpd.apache.org/docs/2.4/ssl/ssl_howto.html) - [Cloudflare SSL/TLS settings](https://developers.cloudflare.com/ssl/) -### Reliability & Monitoring +## Quick Docker Setup + +### Caddy (Easiest) + +```yaml +services: + caddy: + image: caddy:2-alpine + ports: + - "80:80" + - "443:443" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile + - caddy_data:/data + html2rss: + image: html2rss/html2rss-web:latest + environment: + - BASE_URL=https://yourdomain.com + +volumes: + caddy_data: +``` + +`Caddyfile`: + +```caddy +yourdomain.com { + reverse_proxy html2rss:3000 +} +``` + +### Traefik (Auto-discovery) + +```yaml +services: + traefik: + image: traefik:v3.0 + command: + - --providers.docker=true + - --providers.docker.exposedbydefault=false + - --entrypoints.web.address=:80 + - --entrypoints.websecure.address=:443 + - --entrypoints.web.http.redirections.entrypoint.to=websecure + - --entrypoints.web.http.redirections.entrypoint.scheme=https + ports: + - "80:80" + - "443:443" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + html2rss: + image: html2rss/html2rss-web:latest + environment: + - BASE_URL=https://yourdomain.com + labels: + - traefik.enable=true + - traefik.http.routers.html2rss.rule=Host(`yourdomain.com`) + - traefik.http.routers.html2rss.entrypoints=websecure + - traefik.http.routers.html2rss.tls.certresolver=letsencrypt + - traefik.http.services.html2rss.loadbalancer.server.port=3000 +``` + +## Reliability & Monitoring - **Enable auto-updates** using [watchtower](https://containrrr.dev/watchtower/) or similar tools - **Monitor the health check endpoint** (`/health_check.txt`) to detect issues early - **Set up logging** to track errors and performance - **Use environment variables** for configuration instead of hardcoded values -### Performance Optimization +### Example: Adding Watchtower for Auto-updates + +```yaml +services: + watchtower: + image: containrrr/watchtower + volumes: + - /var/run/docker.sock:/var/run/docker.sock + environment: + - WATCHTOWER_CLEANUP=true + - WATCHTOWER_POLL_INTERVAL=300 +``` + +## Performance Optimization - **Configure appropriate resource limits** for your Docker containers - **Use a CDN** for static assets if serving many users - **Monitor memory usage** — html2rss-web can be memory-intensive with large feeds - **Set up caching** for frequently accessed feeds +### Example: Resource Limits + +```yaml +services: + html2rss: + image: html2rss/html2rss-web:latest + deploy: + resources: + limits: + memory: 512M + cpus: "0.5" + reservations: + memory: 256M + cpus: "0.25" +``` + ## Environment Configuration For production, update your environment variables: ```yaml -environment: - RACK_ENV: production - LOG_LEVEL: warn - HEALTH_CHECK_USERNAME: your-secure-username - HEALTH_CHECK_PASSWORD: your-very-secure-password - BASE_URL: https://your-domain.com +services: + html2rss: + image: html2rss/html2rss-web:latest + environment: + RACK_ENV: production + LOG_LEVEL: warn + HEALTH_CHECK_USERNAME: your-secure-username + HEALTH_CHECK_PASSWORD: your-very-secure-password + BASE_URL: https://your-domain.com ``` -Use a [password manager](https://bitwarden.com/password-generator/) to generate strong values. +> 💡 **Security Tip**: Use a [password manager](https://bitwarden.com/password-generator/) to generate strong, unique passwords for all authentication endpoints. ## Share Your Instance @@ -86,6 +186,13 @@ Once your instance is running smoothly: - **Test thoroughly** with various feed types before sharing - **Monitor usage** and be prepared to scale if needed +### Before Going Public + +1. **Test your setup** with different feed configurations +2. **Verify HTTPS** is working correctly +3. **Check performance** under load +4. **Review security settings** and access controls + ## Need Help? - **Deployment issues?** Check our [troubleshooting guide](/troubleshooting/troubleshooting) From 5638092b62bd81d9412e5d7fc563fa086d258595 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 18 Sep 2025 19:38:53 +0200 Subject: [PATCH 03/13] steps --- .../docs/web-application/how-to/deployment.mdx | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/src/content/docs/web-application/how-to/deployment.mdx b/src/content/docs/web-application/how-to/deployment.mdx index 0add7db9..0e2c71b1 100644 --- a/src/content/docs/web-application/how-to/deployment.mdx +++ b/src/content/docs/web-application/how-to/deployment.mdx @@ -3,24 +3,33 @@ title: "Deployment & Production" description: "Deploy html2rss-web to production with Docker. Learn best practices for hosting public instances with security, monitoring, and reliability." --- +import { Steps } from '@astrojs/starlight/components'; + html2rss-web is published on Docker Hub, making it easy to deploy with Docker. The [`docker-compose.yml`](https://github.com/html2rss/html2rss-web/blob/master/docker-compose.yml) from our [Installation Guide](/web-application/getting-started) provides a solid foundation for both development and production use. ## Quick Start -1. Create a directory for your instance: + + +1. **Create a directory** for your instance: ```bash mkdir html2rss-web && cd html2rss-web ``` -2. Download the configuration: + +2. **Download the configuration** files: ```bash curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/docker-compose.yml curl -O https://raw.githubusercontent.com/html2rss/html2rss-web/master/config/feeds.yml ``` -3. Start the instance: + +3. **Start the instance**: ```bash docker compose up -d ``` -4. Open [http://localhost:3000](http://localhost:3000) to verify it’s running. + +4. **Verify it's running** by opening [http://localhost:3000](http://localhost:3000) in your browser. + + > 📖 Need Docker? Follow the [Docker installation guide](https://docs.docker.com/get-docker/) and [Docker Compose install](https://docs.docker.com/compose/install/). From 3d464d238ddf2c4047c78dbdf44619e13dda6ec7 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Tue, 7 Oct 2025 23:48:15 +0300 Subject: [PATCH 04/13] . --- ...-configs.mdx => creating-custom-feeds.mdx} | 110 +++++------ .../docs/get-involved/contributing.mdx | 39 ++-- src/content/docs/getting-started.mdx | 185 ++++++++++++++++++ src/content/docs/index.mdx | 20 +- .../docs/ruby-gem/reference/auto-source.mdx | 8 + 5 files changed, 266 insertions(+), 96 deletions(-) rename src/content/docs/{html2rss-configs.mdx => creating-custom-feeds.mdx} (59%) create mode 100644 src/content/docs/getting-started.mdx diff --git a/src/content/docs/html2rss-configs.mdx b/src/content/docs/creating-custom-feeds.mdx similarity index 59% rename from src/content/docs/html2rss-configs.mdx rename to src/content/docs/creating-custom-feeds.mdx index c8c0782f..48068cd4 100644 --- a/src/content/docs/html2rss-configs.mdx +++ b/src/content/docs/creating-custom-feeds.mdx @@ -1,11 +1,13 @@ --- -title: "Write Your Own Feed Configs" -description: "Step-by-step guide to writing YAML configuration files for custom RSS feeds. Take control when ready-made feeds aren't enough." +title: "Creating Custom Feeds" +description: "Learn to write custom YAML configurations for RSS feeds when auto-sourcing isn't enough." +sidebar: + order: 2 --- -When ready-made feeds aren't enough, you can write your own configuration files to create custom RSS feeds for any website. This guide shows you how to take full control with YAML configs. +When auto-sourcing isn't enough, you can write your own configuration files to create custom RSS feeds for any website. This guide shows you how to take full control with YAML configs. -**Prerequisites:** You should be familiar with the [Web Application](/web-application/getting-started) or [Ruby Gem](/ruby-gem/installation) before diving into custom configurations. +**Prerequisites:** You should be familiar with the [Getting Started](/getting-started) guide before diving into custom configurations. --- @@ -13,7 +15,7 @@ When ready-made feeds aren't enough, you can write your own configuration files **Use custom configs when:** -- **Ready-made feeds don't exist** for the website you want to follow +- **Auto-sourcing doesn't work** for the website you want to follow - **Existing feeds are incomplete** or missing important content - **You need specific formatting** or data extraction - **The website has complex structure** that requires custom selectors @@ -23,27 +25,6 @@ When ready-made feeds aren't enough, you can write your own configuration files --- -## Quick Start - -**Need a feed right now?** - -1. **[Try the web app](/web-application/getting-started)** - No coding required -2. **[Browse ready-made feeds](/feed-directory/)** - Use what others created -3. **Create a config** - When you need custom control (see below) - -**Ready to create a config?** Jump to [Your First Config](#your-first-config). - -### For html2rss-web Users - -If you're using html2rss-web, you can add your config to the `feeds.yml` file: - -1. **Find the example** - Check out the [`example` feed config](https://github.com/html2rss/html2rss-web/blob/master/config/feeds.yml#L9) -2. **Add your config** to the `feeds.yml` file -3. **Test it** by visiting your html2rss-web instance -4. **Debug** using browser developer tools if needed - ---- - ## How It Works A config file is a simple "recipe" that tells html2rss: @@ -87,15 +68,6 @@ This says: "Find each article, get the title from the h2 anchor, and get the lin **Need more details?** Check our [complete guide to selectors](/ruby-gem/reference/selectors/) for all the options. -## Configuration Options - -html2rss-web supports all the same configuration options as the html2rss Ruby gem: - -- **Basic selectors** for title, description, and links -- **Advanced features** like custom headers and dynamic parameters -- **Multiple strategies** for different types of websites -- **Post-processing** to clean up extracted content - --- ## Your First Config @@ -119,41 +91,63 @@ selectors: attribute: href ``` -**Step 3:** Test it with the [web app](/web-application/) or [Ruby gem](/ruby-gem/installation). +**Step 3:** Test it with your html2rss-web instance or the [Ruby gem](/ruby-gem/installation). -**Need help?** See our [detailed tutorial](/ruby-gem/tutorials/your-first-feed/) or [troubleshooting guide](/troubleshooting/troubleshooting) for common issues. +**Need help?** See our [troubleshooting guide](/troubleshooting/troubleshooting) for common issues. --- -## Troubleshooting +## Configuration Options -**Common issues when writing configs:** +html2rss supports many configuration options: -- **No items found?** Check your selectors with browser tools (F12) - the `items.selector` might not match the page structure -- **Invalid YAML?** Use spaces, not tabs, and ensure proper indentation -- **Website not loading?** Check the URL and try accessing it in your browser -- **Missing content?** Some websites load content with JavaScript - you may need to use the `browserless` strategy -- **Wrong data extracted?** Verify your selectors are pointing to the right elements +- **Basic selectors** for title, description, and links +- **Advanced features** like custom headers and dynamic parameters +- **Multiple strategies** for different types of websites +- **Post-processing** to clean up extracted content -**Need more help?** See our [comprehensive troubleshooting guide](/troubleshooting/troubleshooting) or ask in [GitHub Discussions](https://github.com/orgs/html2rss/discussions). +**See our [Ruby Gem Reference](/ruby-gem/reference/)** for complete documentation. --- -## Advanced Features +## Testing Your Config + +**Before sharing your config, test it:** -**Dynamic parameters, custom headers, and more:** See our [advanced features guide](/ruby-gem/how-to/advanced-features/). +1. **Test with Ruby gem:** + ```bash + html2rss feed your-config.yml + ``` + +2. **Test with web app:** Add your config to the `feeds.yml` file and restart your instance + +3. **Check the output:** Make sure all items have titles, links, and descriptions --- -## Contributing +## Sharing Your Config -**Share your config with the community:** +**Help the community by sharing your config:** 1. Go to [html2rss-configs on GitHub](https://github.com/html2rss/html2rss-configs) 2. Click "Fork" → "Add file" → Create `domain.com.yml` 3. Paste your config → "Commit new file" → "Open pull request" -**Need help?** See our [contribution guide](https://github.com/html2rss/html2rss-configs/blob/main/CONTRIBUTING.md). +**Need help?** See our [contribution guide](/get-involved/contributing) for detailed instructions. + +--- + +## Troubleshooting + +**Common issues when writing configs:** + +- **No items found?** Check your selectors with browser tools (F12) - the `items.selector` might not match the page structure +- **Invalid YAML?** Use spaces, not tabs, and ensure proper indentation +- **Website not loading?** Check the URL and try accessing it in your browser +- **Missing content?** Some websites load content with JavaScript - you may need to use the `browserless` strategy +- **Wrong data extracted?** Verify your selectors are pointing to the right elements + +**Need more help?** See our [comprehensive troubleshooting guide](/troubleshooting/troubleshooting) or ask in [GitHub Discussions](https://github.com/orgs/html2rss/discussions). --- @@ -166,23 +160,11 @@ selectors: **For Beginners:** - **[Browse the Feed Directory](/feed-directory/)** - See real-world examples -- **[Try html2rss-web](/web-application/getting-started)** - Create feeds without coding - **[Learn more about selectors](/ruby-gem/reference/selectors/)** - Master CSS selectors +- **[Submit your config via GitHub Web](https://github.com/html2rss/html2rss-configs)** - No Git knowledge required! **For Contributors:** -- **[Submit your config via GitHub Web](https://github.com/html2rss/html2rss-configs)** - No Git knowledge required! - **[Browse existing configs](https://github.com/html2rss/html2rss-configs/tree/master/lib/html2rss/configs)** - See real examples - **[Join discussions](https://github.com/orgs/html2rss/discussions)** - Connect with other users - -**For Developers:** - -- **[Ruby Gem Documentation](/ruby-gem/)** - Full API reference -- **[Advanced Features](/ruby-gem/how-to/advanced-features/)** - Learn advanced techniques -- **[Custom HTTP Requests](/ruby-gem/how-to/custom-http-requests/)** - Handle complex scenarios - -### Need More Help? - -- **[Troubleshooting](/troubleshooting/)** - Get help with common issues -- **[Community Forum](https://github.com/html2rss/html2rss-configs/discussions)** - Ask questions and share ideas -- **[Report Issues](https://github.com/html2rss/html2rss-configs/issues)** - Found a bug? Let us know! +- **[Learn advanced features](/ruby-gem/how-to/advanced-features/)** - Take your configs to the next level diff --git a/src/content/docs/get-involved/contributing.mdx b/src/content/docs/get-involved/contributing.mdx index 8beff244..87ddbf6c 100644 --- a/src/content/docs/get-involved/contributing.mdx +++ b/src/content/docs/get-involved/contributing.mdx @@ -15,44 +15,39 @@ Before you begin, please read our [Code of Conduct](https://github.com/html2rss/ ## How to Contribute -Here are some of the ways you can contribute to the `html2rss` project: +Here are the main ways you can contribute to the `html2rss` project: -### 1. Create a Feed Config +### 1. Create a Feed Config (Most Popular!) Are you missing an RSS feed for a website? You can create your own feed config and share it with the community. It's a great way to get started with `html2rss` and help other users. -The html2rss "ecosystem" is a community project. We welcome contributions of all kinds. This includes new feed configs, suggesting and implementing features, providing bug fixes, documentation improvements, and any other kind of help. +**The easiest way to contribute:** -Which way you choose to add a new feed config is up to you. You can do it manually. Please [submit a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork)! +1. Go to [html2rss-configs on GitHub](https://github.com/html2rss/html2rss-configs) +2. Click "Fork" → "Add file" → Create `domain.com.yml` +3. Paste your config → "Commit new file" → "Open pull request" -After you're done, you can test your feed config by running `bundle exec html2rss feed lib/html2rss/configs//.yml`. +**Need help writing configs?** See our [Creating Custom Feeds](/creating-custom-feeds) guide. -#### Preferred way: manually +**Want to test your config first?** Use the [Ruby gem](/ruby-gem/installation) to test it locally: -1. Fork the `html2rss-config` git repository and run `bundle install` (you need to have Ruby >= 3.3 installed). -2. Create a new folder and file following this convention: `lib/html2rss/configs//.yml` -3. Create the feed config in the `.yml` file. -4. Add this spec file in the `spec/html2rss/configs//_spec.rb` file. - -```ruby - RSpec.describe '/' do - include_examples 'config.yml', described_class - end +```bash +html2rss feed your-config.yml ``` -### 2. Improve this Website +### 2. Host a Public Instance -This website is built with Astro and Starlight and is hosted on GitHub Pages. If you have any ideas for improving the documentation or the design, we'd love to hear from you. +The [`html2rss-web`](https://github.com/html2rss/html2rss-web) project is a web application that provides RSS feeds with stable HTTPS URLs. You can host your own public instance to help other users access feeds. -[**Find the source code on GitHub**](https://github.com/html2rss/html2rss.github.io) +[**Learn how to host a public instance**](/web-application/how-to/deployment) -### 3. Host a Public Instance +### 3. Improve the Website -The [`html2rss-web`](https://github.com/html2rss/html2rss-web) project is a web application that allows you to create and manage your RSS feeds through a user-friendly interface. You can host your own public instance to help other users create feeds. +This website is built with Astro and Starlight and is hosted on GitHub Pages. If you have any ideas for improving the documentation or the design, we'd love to hear from you. -[**Learn how to host a public instance**](/web-application/how-to/deployment) +[**Find the source code on GitHub**](https://github.com/html2rss/html2rss.github.io) -### 4. Improve the `html2rss` Gem +### 4. Improve the `html2rss` Engine Are you a Ruby developer? You can help us improve the core `html2rss` gem. Whether you're fixing a bug, adding a new feature, or improving the documentation, your contributions are welcome. diff --git a/src/content/docs/getting-started.mdx b/src/content/docs/getting-started.mdx new file mode 100644 index 00000000..20eaf93c --- /dev/null +++ b/src/content/docs/getting-started.mdx @@ -0,0 +1,185 @@ +--- +title: "Getting Started" +description: "Learn how to get RSS feeds from any website. Start with existing feeds or create your own in minutes." +sidebar: + order: 1 +--- + +Ready to get RSS feeds from any website? Choose your path below. No technical knowledge required! + +## Quick Start Options + +### Option 1: Use a Public Instance (Easiest) + +**Try html2rss right now:** + +1. **Go to a public html2rss instance** (see our [community wiki](https://github.com/html2rss/html2rss-web/wiki/Instances) for available instances) +2. **Browse existing feeds** or create auto-sourced feeds from any URL +3. **Copy the RSS URL** and add it to your feed reader +4. **Done!** You're now following the website in your RSS reader + +**Perfect for:** Testing, quick feeds, or if you don't want to install anything. + +### Option 2: Install Your Own Instance + +**For full control and custom feeds:** + +1. **Follow the installation steps below** +2. **Create auto-sourced feeds** from any website +3. **Add custom configs** when you need more control +4. **Run your own RSS feed server** with stable HTTPS URLs + +**Perfect for:** Power users, custom configurations, or when you need reliability. + +--- + +## Installation Guide + +This guide will help you set up your own copy of html2rss-web on your computer. Don't worry - we'll walk you through every step! + +### What You'll Need + +- **Docker** - A tool that makes installation simple (like an app store for server software) +- **About 10 minutes** - The whole process is quick and automated + +**Don't have Docker?** [Install it first](https://docs.docker.com/get-started/) - it's free and works on all major operating systems. + +### Step 1: Create a Folder + +Create a new folder on your computer to store html2rss-web files: + +**Create a new folder** on your computer and name it "html2rss-web". You can do this through your file manager or terminal: + +```bash +mkdir html2rss-web +cd html2rss-web +``` + +### Step 2: Create the Configuration File + +Create a file called `docker-compose.yml` in your new folder. This file tells Docker how to set up html2rss-web with all the features you need. + +**How to create the file:** + +- **Using a text editor:** Create a new file and save it as `docker-compose.yml` +- **Using terminal:** Use any text editor to create the file + +```yaml +services: + html2rss-web: + image: gilcreator/html2rss-web + restart: unless-stopped + ports: + - "127.0.0.1:3000:3000" + volumes: + - type: bind + source: ./feeds.yml + target: /app/config/feeds.yml + read_only: true + environment: + RACK_ENV: production + HEALTH_CHECK_USERNAME: health + HEALTH_CHECK_PASSWORD: please-set-YOUR-OWN-veeeeeery-l0ng-aNd-h4rd-to-gue55-Passw0rd! + BROWSERLESS_IO_WEBSOCKET_URL: ws://browserless:3001 + BROWSERLESS_IO_API_TOKEN: 6R0W53R135510 + + watchtower: + image: containrrr/watchtower + restart: unless-stopped + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - "~/.docker/config.json:/config.json" + command: --cleanup --interval 7200 + + browserless: + image: "ghcr.io/browserless/chromium" + restart: unless-stopped + ports: + - "127.0.0.1:3001:3001" + environment: + PORT: 3001 + CONCURRENT: 10 + TOKEN: 6R0W53R135510 +``` + +### Step 3: Download the Feed List + +html2rss-web needs a list of feeds to work with. Download our pre-made list: + +**Download the feeds.yml file:** + +- **Using your browser:** Right-click [this link](https://raw.githubusercontent.com/html2rss/html2rss-web/master/config/feeds.yml) → Save As → Name it "feeds.yml" → Save in your html2rss-web folder +- **Using terminal:** Open Terminal in your html2rss-web folder and run: + +```bash +curl https://raw.githubusercontent.com/html2rss/html2rss-web/master/config/feeds.yml -o feeds.yml +``` + +### Step 4: Start html2rss-web + +Now start html2rss-web: + +**Start html2rss-web:** + +Open a terminal in your html2rss-web folder and run: + +```bash +docker compose up -d +``` + +**That's it!** 🎉 html2rss-web is now running. + +**To verify it's working:** + +1. Open your web browser +2. Go to `http://localhost:3000` +3. You should see the html2rss-web interface with a list of available feeds + +**If you see the interface, congratulations!** You've successfully set up html2rss-web. + +--- + +## Using Your Instance + +### Browse Existing Feeds + +Your html2rss-web instance comes with pre-configured feeds for popular websites. Browse the list and copy any RSS URL to your feed reader. + +### Create Auto-Sourced Feeds + +**Want a feed for a website that's not in the list?** + +1. **Go to your html2rss-web interface** (http://localhost:3000) +2. **Enter any website URL** you want to create a feed for +3. **Click "Generate Feed"** - html2rss will automatically analyze the page +4. **Copy the RSS URL** and add it to your feed reader + +**This works for most websites** without any configuration needed! + +### Add Custom Feeds + +**Need more control?** You can add custom feed configurations to your `feeds.yml` file. See our [Creating Custom Feeds](/creating-custom-feeds) guide for details. + +--- + +## Troubleshooting + +**Common issues when getting started:** + +- **Instance won't start?** Check that Docker is running and the port isn't already in use +- **Can't access the interface?** Make sure you're going to `http://localhost:3000` +- **Auto-sourced feed not working?** Some websites need custom configuration - see our [troubleshooting guide](/troubleshooting/troubleshooting) +- **Need help?** Join our [community discussions](https://github.com/orgs/html2rss/discussions) + +--- + +## Next Steps + +**Ready for more?** + +- **[Browse the Feed Directory](/feed-directory/)** - See real-world examples +- **[Create Custom Feeds](/creating-custom-feeds)** - Learn to write your own configurations +- **[Troubleshooting Guide](/troubleshooting/troubleshooting)** - Solve common issues +- **[Join the Community](https://github.com/orgs/html2rss/discussions)** - Get help and share ideas + +**🎉 Congratulations!** You now have your own RSS feed server running. diff --git a/src/content/docs/index.mdx b/src/content/docs/index.mdx index 9ea27341..adfe355d 100644 --- a/src/content/docs/index.mdx +++ b/src/content/docs/index.mdx @@ -7,7 +7,7 @@ Ever wished you could follow your favorite websites like a social media feed? Th ## 🚀 Get Started in 30 Seconds -**Try it now:** [Browse Ready-Made Feeds](/feed-directory/) | [Create Your First Feed](/web-application/getting-started) +**Try it now:** [Browse Ready-Made Feeds](/feed-directory/) | [Get Started](/getting-started) --- @@ -38,8 +38,8 @@ html2rss is a user-friendly tool that turns any website into an RSS feed. Think **Two ways to use html2rss:** -- **🌐 Web App** - Point, click, done! Perfect for everyone -- **⚙️ Ruby Gem** - For developers who want to code their own feeds +- **🌐 Web App** - RSS feed server with auto-sourcing. Perfect for everyone +- **⚙️ Ruby Gem** - The core engine that powers everything --- @@ -47,21 +47,21 @@ html2rss is a user-friendly tool that turns any website into an RSS feed. Think ### I'm Getting Started with RSS -1. **[See Examples First](/feed-directory/)** - Browse 100+ ready-made feeds -2. **[Try the Web App](/web-application/getting-started)** - Create feeds without coding +1. **[Get Started](/getting-started)** - Complete guide to using html2rss +2. **[Browse Examples](/feed-directory/)** - See 100+ ready-made feeds 3. **[Learn More](/about/)** - Understand how it all works ### I Want to Create Custom Feeds -1. **[Web App](/web-application/getting-started)** - Start with the easy interface -2. **[Ruby Gem](/ruby-gem/installation)** - For advanced customization -3. **[Config Guide](/html2rss-configs/)** - Learn to write your own configs +1. **[Creating Custom Feeds](/creating-custom-feeds)** - Learn to write your own configs +2. **[Ruby Gem Reference](/ruby-gem/)** - Full technical documentation +3. **[Share Your Configs](/get-involved/contributing/)** - Help the community ### I'm a Developer -1. **[Ruby Gem Docs](/ruby-gem/)** - Full API reference +1. **[Ruby Gem Reference](/ruby-gem/)** - Full API documentation 2. **[Advanced Features](/ruby-gem/how-to/advanced-features/)** - Custom HTTP requests, etc. -3. **[Contribute](/get-involved/contributing/)** - Help improve the project +3. **[Contribute to Core](/get-involved/contributing/)** - Help improve the engine --- diff --git a/src/content/docs/ruby-gem/reference/auto-source.mdx b/src/content/docs/ruby-gem/reference/auto-source.mdx index 654608dc..ee0f2bd8 100644 --- a/src/content/docs/ruby-gem/reference/auto-source.mdx +++ b/src/content/docs/ruby-gem/reference/auto-source.mdx @@ -20,6 +20,12 @@ auto_source: {} 1. **`schema`:** Parses `