From 943cc1265aa5717a7850ce37e307586b6d699b81 Mon Sep 17 00:00:00 2001 From: Carter Medlin Date: Mon, 1 Nov 2021 10:54:21 -0700 Subject: [PATCH 1/2] Separating readme files --- README.md | 121 ++-------------------------------- wordpress-to-github/README.md | 104 +---------------------------- 2 files changed, 5 insertions(+), 220 deletions(-) diff --git a/README.md b/README.md index f1d865b..adac475 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,6 @@ -# wordpress-to-github +# wordpress-to-github development project -A service for taking content from Wordpress instances and pushing the HTML content and JSON meta to Github. - -## How does it work? - -The service scans a list of WordPress projects. Each project's WordPress site is queried through the WordPress API. The service compares the objects from the API to the project's target GitHub branch. Content changes are recorded in GitHub as commits. - -### Caching - -The service will ask WordPress for object counts and last updated timestamps (and cache them) to determine if it should run. If you are working on the WordPress API output without updating the content itself, changes may not appear until the cache is reset. +The development project for the wordpress-to-github NPM module [NPM module](https://www.npmjs.com/package/@cagov/wordpress-to-github). The readme for the development project is [here](https://github.com/cagov/wordpress-to-github/tree/main/wordpress-to-github#readme). ## Components @@ -24,112 +16,7 @@ The the trigger service is set to `"authLevel": "function"`, so Azure FaaS Funct ### wordpress-to-github -The core features that will eventually be extracted to be an NPM package. - -## Config files - -There are a few configuration files that need to be used. - -### wordpress-to-github.config.json - -Controls how the service will place content in GitHub. - -```json -{ - "$schema": "https://raw.githubusercontent.com/cagov/wordpress-to-github/main/wordpress-to-github/schemas/wordpress-to-github.config.schema.json", - "meta": { - "title": "wordpress-to-github endpoints config file", - "description": "wordpress-to-github endpoints config file" - }, - "data": { - "disabled": false, - "PostPath": "wordpress/posts", - "PagePath": "wordpress/pages", - "MediaPath": "wordpress/media", - "GeneralFilePath": "wordpress/general/general.json", - "ExcludeProperties": ["content", "_links"] - } -} -``` - -`disabled` -: Set to true to disable processing for this project. - -`PostPath` -: Where should the posts go? - -`PagePath` -: Where should the pages go? - -`MediaPath` -: Where should image media go? - -`GeneralFilePath` -: The full path and filename for a `general.json` file that contains information about the whole site. - -`ExcludeProperties` -: Which WordPress properties should we suppress in output? - -### endpoints.json - -Contains the projects to process with the service. - -```json -{ - "$schema": "./endpoints.schema.json", - "meta": { - "title": "endpoints config file", - "description": "endpoints config file" - }, - "data": { - "projects": [ - { - "name": "drought.ca.gov", - "description": "Drought production website", - "enabled": true, - "enabledLocal": false, - "ReportingChannel_Slack": "C1234567890", - "WordPressSource": { - "url": "https://live-drought-ca-gov.pantheonsite.io", - "tags_exclude": ["staging", "development"] - }, - "GitHubTarget": { - "Owner": "cagov", - "Repo": "drought.ca.gov", - "Branch": "main", - "ConfigPath": "wordpress/wordpress-to-github.config.json" - } - } - ] - } -} -``` - -`name` : Friendly name for this job when it runs locally. - -`description` : Describe what this is being used for in this endpoint configuration. - -`enabled` : Should we process this endpoint? - -`enabledLocal` : Should we process this endpoint when running in local development? - -`ReportingChannel_Slack` : Slack channel to report activity to. - -`WordPressSource` : Describes the Wordpress instance to read from. - -`url` : URL of the Wordpress instance to read from. - -`tags_exclude` : Ignore Pages/Posts with these tags (Case sensitive!). - -`GitHubTarget` : The endpoint target to deploy changes. - -`Owner` : GitHub Owner. - -`Repo` : GitHub Repo. - -`Branch` : GitHub Target Branch. - -`ConfigPath` : Path to config.json file for this endpoint. +The development package for the wordpress-to-github [NPM module](https://www.npmjs.com/package/@cagov/wordpress-to-github). ## Setting up Local Execution/Debugging @@ -216,4 +103,4 @@ With the list of keys you can... - `Renew key value` to invalidate an old key for replacement - 🗑️ Delete a key that should no longer be used -Make sure to name the key appropriately, and have a unique key per consuming site. Using the domain name as the key name is a good idea. +Make sure to name the key appropriately, and have a unique key per consuming site. Using the domain name as the key name is a good idea. diff --git a/wordpress-to-github/README.md b/wordpress-to-github/README.md index 7900774..5479176 100644 --- a/wordpress-to-github/README.md +++ b/wordpress-to-github/README.md @@ -1,6 +1,6 @@ # wordpress-to-github -A service for taking content from Wordpress instances and pushing the HTML content and JSON meta to Github. +A module for taking content from Wordpress instances and pushing the HTML content and JSON meta to Github. ## How does it work? @@ -10,22 +10,6 @@ The service scans a list of WordPress projects. Each project's WordPress site is The service will ask WordPress for object counts and last updated timestamps (and cache them) to determine if it should run. If you are working on the WordPress API output without updating the content itself, changes may not appear until the cache is reset. -## Components - -### WordpressSync - -An Azure Function as as Service (FaaS) site that operates on a Cron schedule to poll Wordpress sites for processing. - -### WordpressSyncHttpTrigger - -An Azure Function as as Service (FaaS) site that can be the target for update notifications. Wordpress sites can be configured to hit this service when changes are made to trigger updates without waiting for the polling service. - -The the trigger service is set to `"authLevel": "function"`, so Azure FaaS Function Keys need to be maintained for trigger operations to work. See below for how to update them. - -### wordpress-to-github - -The core features that will eventually be extracted to be an NPM package. - ## Config files There are a few configuration files that need to be used. @@ -130,89 +114,3 @@ Contains the projects to process with the service. `Branch` : GitHub Target Branch. `ConfigPath` : Path to config.json file for this endpoint. - -## Setting up Local Execution/Debugging - -When using Visual Studio Code, you can run the polling service locally. Only projects with `enabledLocal: true` will run. It is recommended that you keep all projects set to `enabledLocal: false` until you are sure you want to run them. The `RUN AND DEBUG` launch menu in VS Code should contain `Debug DIRECT WordpressSync`; use that to run locally with debugging. - -### `local.settings.json` - -You will need to define a `local.settings.json` file in the project root with the following options. - -```json -{ - ... - "Values": { - ... - "GITHUB_NAME": "...Your GitHub Name...", - "GITHUB_EMAIL": "...Your GitHub Email...", - "GITHUB_TOKEN": "...Your GitHub API Token...", - "SLACKBOT_TOKEN":"...Your Slackbot API Token..." - ... - } -} -``` - -`GITHUB_NAME` : The name that will appear on commits. - -`GITHUB_EMAIL` : The email that will appear on commits. - -`GITHUB_TOKEN` : Your token used to authenticate with GitHub. Get one [here](https://github.com/settings/tokens). - -`SLACKBOT_TOKEN` : Your token used to authenticate with your Slack app. Make one [here](https://api.slack.com/apps/). - -### Local running with select projects - -Running this project locally is a good way to see exactly what it is querying, what it decides to update and what commits it creates. - -The config file `WordpressSync/endpoints.json` contains the list of all endpoints to be reviewed and which repos to push content updates to. The enabledLocal on every one of these is set to false by default. - -If you run this project locally in VSCode while all the endpoints have `"enabledLocal": false`, it will stop execution after reviewing that file with a message that you need to enable at least one endpoint to make updates. - -Execute this locally in VSCode by: - -- Selecting the debug section in the sidebar (triangle with bug icon) -- Choosing `Debug DIRECT WordpressSync` from the `RUN AND DEBUG` pulldown menu at the top of the right sidebar -- Clicking the triangle next to your pulldown selection to begin execution - -Executing the code on a live endpoint will behave exactly as this does in production, it will check for updates at the server endpoint, create a commit of these changes to the target repository specified in `endpoints.json` and send slack notifications to the channel specified. - -These commits will be performed with the account associated with your github token in your `local.settings.json` file. The `local.settings.json` file is gitignored and never committed to the repository. If it were accidentally committed github would immediately invalidate the exposed token and notify the committer. Individuals without access to accounts with permission to commit directly to the associated repo will not be able to create tokens the service can use to commit changes. - -Slack messages can be redirected to your own debug channels by changing the channel id or disabled by removing the `ReportingChannel_Slack` value from the project which you have `"enabledLocal": true`. - -When an endpoint is enabled it will be queried for each supported WordPress object type (Post/Page/Media), the VSCode command line console will show the running function's log output which will list the endpoints it is reviewing for updates. - -If an update is discovered the service will create a commit and logs will print a link to the commit created on github. - -If no update is discovered no further output is displayed. - -This polling service is executing in its production FaaS instance on a 2 minute timer. Running it locally will be performing the same actions occuring constantly in production against the same endpoints and github repos defined in the endpoints.json file. - -### Local running just like the real instance - -You can run all projects just like the real instance, by clearing the environment debug setting in `WordpressSync\debug.js`. - -```js -process.env.debug = true; //set to false or remove to run like the real instance -``` - -`enabledLocal` is ignored using this method. The `enabled` setting in each project is used, just like the real running instance. - -## Setting Trigger Function Keys - -Function keys need to be kept secret. They are required to authenticate to the function service from a WordPress notification instance. The keys are added to the end of the URL (as `&code=`) that will POST to the `WordpressSyncHttpTrigger` function. `WordpressSyncHttpTrigger` runs in `"authLevel": "function"` mode that requires Function Keys to operate. - -To view the keys. - -1. As an Admin, go the the Function App in Azure. -1. On the side bar, select `Functions`. -1. In the list of functions, select `WordpressSyncHttpTrigger`. -1. On the side bar, select `Function Keys`. - -With the list of keys you can... - -- View an existing key by clicking `Hidden value. Click to show value` -- Create a `New function key` for assigning to a new WordPress notification instance -- `Renew key value` to invalidate an old key for replacement -- 🗑️ Delete a key that should no longer be used From 128ddf71bca0fd261456c419ae58adecfc8fec2a Mon Sep 17 00:00:00 2001 From: Carter Medlin Date: Tue, 2 Nov 2021 10:19:29 -0700 Subject: [PATCH 2/2] Added sample output --- wordpress-to-github/README.md | 258 ++++++++++++++++++++++++++++++++++ 1 file changed, 258 insertions(+) diff --git a/wordpress-to-github/README.md b/wordpress-to-github/README.md index 5479176..c58a418 100644 --- a/wordpress-to-github/README.md +++ b/wordpress-to-github/README.md @@ -114,3 +114,261 @@ Contains the projects to process with the service. `Branch` : GitHub Target Branch. `ConfigPath` : Path to config.json file for this endpoint. + +## Sample output + +### `general / general.json` + +This file will contain global information for the WordPress istance + +```json +{ + "meta": { + "api_version": "v2", + "api_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/", + "object_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json?_fields=description,gmt_offset,name,namespaces,timezone_string,home,url", + "process": { + "source_code": "https://github.com/cagov/wordpress-to-github", + "source_data": "https://as-go-covid19-d-001.azurewebsites.net", + "deployment_target": "https://github.com/cagov/automation-development-target/tree/main" + }, + "refresh_frequency": "as needed" + }, + "data": { + "name": "COVID-19", + "description": "Content site for COVID-19", + "url": "http://as-go-covid19-d-001.azurewebsites.net", + "home": "https://as-go-covid19-d-001.azurewebsites.net", + "gmt_offset": "0", + "timezone_string": "", + "namespaces": [ + "oembed/1.0", + "wp/v2", + "wp-site-health/v1" + ] + } +} +``` + +### `posts / [slug].json` + +All posts .son files will appear next to their .html content files. + +```json +{ + "meta": { + "created_date": "2020-04-13T19:51:57", + "updated_date": "2020-07-14T22:32:38", + "field_reference": "https://developer.wordpress.org/rest-api/reference/posts/", + "api_version": "v2", + "api_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/", + "object_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/posts/1196", + "process": { + "source_code": "https://github.com/cagov/wordpress-to-github", + "source_data": "https://as-go-covid19-d-001.azurewebsites.net", + "deployment_target": "https://github.com/cagov/automation-development-target/tree/main" + }, + "refresh_frequency": "as needed" + }, + "data": { + "id": 1196, + "date": "2020-04-13T19:51:57", + "date_gmt": "2020-04-13T19:51:57", + "guid": "http://as-go-covid19-d-001.azurewebsites.net/?p=1196", + "modified": "2020-07-14T22:32:38", + "modified_gmt": "2020-07-14T22:32:38", + "slug": "404", + "status": "publish", + "type": "post", + "link": "https://as-go-covid19-d-001.azurewebsites.net/2020/04/13/404/", + "title": "404", + "excerpt": "

We couldn’t find this page. Want to search? No pudimos encontrar esta página. Chúng tôi không thể tìm thấy trang này. لم نتمكن من العثور على هذه الصفحة. 이 페이지를 찾을 수 없습니다. 我们找不到此页面。

\n", + "author": "Carter Medlin", + "featured_media": 0, + "comment_status": "closed", + "ping_status": "open", + "sticky": false, + "template": "", + "format": "standard", + "meta": [], + "categories": [ + "error page" + ], + "tags": [ + "do-not-crawl", + "machine-translated" + ], + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/2020/04/13/404/" + } +} +``` + +### `pages / [slug].json` + +Pages work like posts, with some differences. + +```json +{ + "meta": { + "created_date": "2021-02-03T21:44:55", + "updated_date": "2021-02-03T21:44:55", + "field_reference": "https://developer.wordpress.org/rest-api/reference/pages/", + "api_version": "v2", + "api_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/", + "object_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/pages/8932", + "process": { + "source_code": "https://github.com/cagov/wordpress-to-github", + "source_data": "https://as-go-covid19-d-001.azurewebsites.net", + "deployment_target": "https://github.com/cagov/automation-development-target/tree/main" + }, + "refresh_frequency": "as needed" + }, + "data": { + "id": 8932, + "date": "2021-02-03T21:44:55", + "date_gmt": "2021-02-03T21:44:55", + "guid": "http://as-go-covid19-d-001.azurewebsites.net/?page_id=8932", + "modified": "2021-02-03T21:44:55", + "modified_gmt": "2021-02-03T21:44:55", + "slug": "home", + "status": "publish", + "type": "page", + "link": "https://as-go-covid19-d-001.azurewebsites.net/", + "title": "Home", + "excerpt": "

homepage

\n", + "author": "Aaron Hans", + "featured_media": 0, + "parent": 0, + "menu_order": 0, + "comment_status": "closed", + "ping_status": "closed", + "template": "", + "meta": [], + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/" + } +} +``` + +### `media / [year] / [month] / [slug].json` + +Media files include sizes. + +```json +{ + "meta": { + "created_date": "2020-07-14T18:55:54", + "updated_date": "2020-07-14T18:55:54", + "field_reference": "https://developer.wordpress.org/rest-api/reference/pages/", + "api_version": "v2", + "api_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/", + "object_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-json/wp/v2/media/4826", + "process": { + "source_code": "https://github.com/cagov/wordpress-to-github", + "source_data": "https://as-go-covid19-d-001.azurewebsites.net", + "deployment_target": "https://github.com/cagov/automation-development-target/tree/main" + }, + "refresh_frequency": "as needed" + }, + "data": { + "id": 4826, + "date": "2020-07-14T18:55:54", + "date_gmt": "2020-07-14T18:55:54", + "guid": "/wp-content/uploads/2020/07/15485_lores.jpg", + "modified": "2020-07-14T18:55:54", + "modified_gmt": "2020-07-14T18:55:54", + "slug": "15485_lores", + "status": "inherit", + "type": "attachment", + "link": "https://as-go-covid19-d-001.azurewebsites.net/2020/05/11/masks-and-ppe/15485_lores/", + "title": "15485_lores", + "author": "Aaron Hans", + "comment_status": "closed", + "ping_status": "closed", + "template": "", + "meta": [], + "description": "

\"\"

\n", + "caption": "", + "alt_text": "", + "media_type": "image", + "mime_type": "image/jpeg", + "media_details": { + "width": 700, + "height": 466, + "file": "2020/07/15485_lores.jpg", + "sizes": { + "medium": { + "file": "15485_lores-300x200.jpg", + "width": 300, + "height": 200, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores-300x200.jpg" + }, + "thumbnail": { + "file": "15485_lores-150x150.jpg", + "width": 150, + "height": 150, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores-150x150.jpg" + }, + "full": { + "file": "15485_lores.jpg", + "width": 700, + "height": 466, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores.jpg" + } + }, + "image_meta": { + "aperture": "0", + "credit": "", + "camera": "", + "caption": "", + "created_timestamp": "0", + "copyright": "", + "focal_length": "0", + "iso": "0", + "shutter_speed": "0", + "title": "", + "orientation": "0", + "keywords": [] + } + }, + "post": 2997, + "source_url": "/wp-content/uploads/2020/07/15485_lores.jpg", + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-content/uploads/2020/07/15485_lores.jpg", + "sizes": [ + { + "type": "full", + "path": "2020/07/15485_lores.jpg", + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-content/uploads/2020/07/15485_lores.jpg", + "file": "15485_lores.jpg", + "width": 700, + "height": 466, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores.jpg" + }, + { + "type": "medium", + "path": "2020/07/15485_lores-300x200.jpg", + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-content/uploads/2020/07/15485_lores-300x200.jpg", + "file": "15485_lores-300x200.jpg", + "width": 300, + "height": 200, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores-300x200.jpg" + }, + { + "type": "thumbnail", + "path": "2020/07/15485_lores-150x150.jpg", + "wordpress_url": "https://as-go-covid19-d-001.azurewebsites.net/wp-content/uploads/2020/07/15485_lores-150x150.jpg", + "file": "15485_lores-150x150.jpg", + "width": 150, + "height": 150, + "mime_type": "image/jpeg", + "source_url": "/wp-content/uploads/2020/07/15485_lores-150x150.jpg" + } + ], + "path": "2020/07/15485_lores.jpg" + } +} +``` \ No newline at end of file