Skip to content

Commit

Permalink
[FEATURE] Introduce Config API
Browse files Browse the repository at this point in the history
  • Loading branch information
eliashaeussler committed Feb 22, 2024
1 parent 0442a60 commit 9793e8f
Show file tree
Hide file tree
Showing 17 changed files with 1,266 additions and 58 deletions.
101 changes: 101 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ The following input parameters are available:
|-------------------------------------------------|------------------------------------------------------------------------|
| [`sitemaps`](#sitemaps) | URLs or local filenames of XML sitemaps to be warmed up |
| [`--allow-failures`](#--allow-failures) | Allow failures during URL crawling and exit with zero |
| [`--config`](#--config) | Path to configuration file |
| [`--crawler-options`, `-o`](#--crawler-options) | JSON-encoded string of additional config for configurable crawlers |
| [`--crawler`, `-c`](#--crawler) | FQCN of the crawler to use for cache warmup |
| [`--exclude`, `-e`](#--exclude) | Patterns of URLs to be excluded from cache warmup |
Expand Down Expand Up @@ -130,6 +131,31 @@ $ cache-warmup -u "https://www.example.org/" -u "https://www.example.org/de/"
| Multiple values allowed ||
| Default | **** |

#### `--config`

Path to a configuration file. Read more in the
[Configuration file](#configuration-file) section below.

At the moment, the following file formats are available:

* [`json`](#json-and-yaml)
* [`php`](#php)
* [`yaml`/`yml`](#json-and-yaml)

> [!NOTE]
> Configuration options provided by command options take
> precedence over those provided by configuration files.
```bash
$ cache-warmup --config cache-warmup.yaml
```

| Shorthand ||
|:------------------------|:------|
| Required | **** |
| Multiple values allowed | **** |
| Default | **** |

#### `--exclude`

Patterns of URLs to be excluded from cache warmup.
Expand Down Expand Up @@ -533,6 +559,81 @@ object. It includes the following properties:

💡 The complete JSON structure can be found in the provided [JSON schema][22].

### Configuration file

Instead of passing all available configuration options as
command parameters, it is also possible to provide a
configuration file to be used for cache warmup. At the moment,
the following file formats are supported:

* [`json`](#json-and-yaml)
* [`php`](#php)
* [`yaml`/`yml`](#json-and-yaml)

#### JSON and YAML

For JSON and YAML files, the name of each configuration option
can be derived from the list of available
[command parameters](#command-line-usage) and must be written in
camel case, e.g. `crawler-options` is configured as `crawlerOptions`.

> [!NOTE]
> Crawler options must be written in object notation instead of
> JSON notation (see examples below).
JSON example:

```json
{
"sitemaps": [
"https://www.example.org/sitemap.xml"
],
"crawlerOptions": {
"concurrency": 3,
"request_options": {
"delay": 3000
}
}
}
```

YAML example:

```yaml
sitemaps:
- https://www.example.org/sitemap.xml
crawlerOptions:
concurrency: 3
request_options:
delay: 3000
```

> [!TIP]
> For a full list of supported configuration options, have a look at the
> [`EliasHaeussler\CacheWarmup\Config\CacheWarmupConfig`](src/Config/CacheWarmupConfig.php)
> class.
#### PHP

PHP config files must return a closure which returns an instance
of [`EliasHaeussler\CacheWarmup\Config\CacheWarmupConfig`](src/Config/CacheWarmupConfig.php).

Example:

```php
use EliasHaeussler\CacheWarmup;

return static function (CacheWarmup\Config\CacheWarmupConfig $config) {
$config->addSitemap('https://www.example.org/sitemap.xml');
$config->setCrawlerOption('concurrency', 3);
$config->setCrawlerOption('request_options', [
'delay' => 3000,
]);

return $config;
};
```

## 🧑‍💻 Contributing

Please have a look at [`CONTRIBUTING.md`](CONTRIBUTING.md).
Expand Down
3 changes: 2 additions & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@
"psr/http-message": "^1.0 || ^2.0",
"psr/log": "^2.0 || ^3.0",
"symfony/console": "^5.4 || ^6.0 || ^7.0",
"symfony/filesystem": "^5.4 || ^6.0 || ^7.0"
"symfony/filesystem": "^5.4 || ^6.0 || ^7.0",
"symfony/yaml": "^5.4 || ^6.0 || ^7.0"
},
"require-dev": {
"armin/editorconfig-cli": "^1.8 || ^2.0",
Expand Down
74 changes: 73 additions & 1 deletion composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 1 addition & 18 deletions src/CacheWarmer.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@
use GuzzleHttp\Client;
use GuzzleHttp\ClientInterface;
use GuzzleHttp\Exception\GuzzleException;
use GuzzleHttp\Psr7;
use Psr\Http\Message;

use function array_key_exists;
use function array_values;
Expand All @@ -36,7 +34,6 @@
use function is_array;
use function is_string;
use function preg_match;
use function str_contains;
use function str_starts_with;

/**
Expand Down Expand Up @@ -124,8 +121,7 @@ public function addSitemaps(array|string|Sitemap\Sitemap $sitemaps): self
foreach ($sitemaps as $sitemap) {
// Parse sitemap URL to valid sitemap object
if (is_string($sitemap)) {
$sitemapUri = $this->resolveSitemapUri($sitemap);
$sitemap = new Sitemap\Sitemap($sitemapUri);
$sitemap = Sitemap\Sitemap::createFromString($sitemap);
}

// Throw exception if sitemap is invalid
Expand Down Expand Up @@ -196,19 +192,6 @@ public function addUrl(string|Sitemap\Url $url): self
return $this;
}

private function resolveSitemapUri(string $sitemap): Message\UriInterface
{
// Sitemap is a remote URL
if (str_contains($sitemap, '://')) {
return new Psr7\Uri($sitemap);
}

// Sitemap is a local file
$file = Helper\FilesystemHelper::resolveRelativePath($sitemap);

return new Psr7\Uri('file://'.$file);
}

private function exceededLimit(): bool
{
return $this->limit > 0 && count($this->urls) >= $this->limit;
Expand Down
Loading

0 comments on commit 9793e8f

Please sign in to comment.