Syndicate Hugo blog posts to dev.to and Hashnode#4872
Merged
shai-almog merged 17 commits intomasterfrom May 6, 2026
Merged
Conversation
Daily GitHub Action that picks the oldest blog post under docs/website/content/blog dated after 2026-04-30, at least 7 days old, and not yet syndicated to a given platform. The script absolutizes relative links/images, inserts a one-sentence "What is Codename One" blurb after the fold, and POSTs to each platform with canonical_url pointing back to the original on www.codenameone.com. Per-platform state in scripts/website/syndication-state.json so partial failures retry only the failed side. Requires repo secrets: DEVTO_API_KEY, HASHNODE_TOKEN, HASHNODE_PUBLICATION_ID. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Contributor
Cloudflare Preview
|
Adds foojay.io as a third syndication target. Unlike dev.to and Hashnode the foojay flow creates a WP draft via /wp-json/wp/v2/posts so the foojay editors can review before publishing. The canonical link is surfaced as a visible note at the top of the draft (rather than a meta field) so the reviewer can wire it up using whichever SEO plugin foojay runs. Side effects: - platforms with missing credentials are now skipped at startup with a note instead of failing the whole run, so adding a new platform later does not strand the candidate selector - requests now send a real User-Agent and Accept header (Cloudflare in front of foojay rejected the default Python-urllib UA with error 1010) - foojay credentials (FOOJAY_USER / FOOJAY_PASSWORD) wired through the workflow as optional secrets; the script auto-skips foojay until both are configured Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
foojay runs Yoast SEO, so the canonical URL is now sent as meta._yoast_wpseo_canonical on the WP draft. Yoast registers that key as a REST-exposed post meta, so the standard /wp-json/wp/v2/posts payload carries it through. The visible "originally published" line at the top of the draft body is dropped — Yoast handles the SEO directive and the "What is Codename One" blurb still provides reader-facing attribution. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
foojay.io has Wordfence configured to disable WordPress Application Passwords, so there is no usable Basic Auth path for the WP REST API from the syndication script. Removing the foojay code path until / unless foojay editorial offers an alternative auth method (JWT, per-user API key, etc.). The User-Agent header and skip-when-unconfigured behaviour introduced alongside the foojay work are kept — they are useful for the remaining platforms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds scripts/website/syndicate_browser_posts.py — a Playwright-based counterpart to the API syndicator. Each target site has its own adapter (login + draft submission). State and post selection are shared with the API script via syndication-state.json, so a post is "candidate" until all configured platforms — API and browser — have a record. Adapters: - foojay: hybrid path. Playwright drives wp-login.php to obtain a real session (Wordfence has Application Passwords disabled, so token auth is out), then the script POSTs the draft via /wp-json/wp/v2/posts using the session cookies + X-WP-Nonce. Pure UI submission was attempted but Cloudflare in front of foojay challenges form POSTs and drops the payload, so drafts never landed. Yoast canonical isn't REST-writable on this Yoast install, so the canonical is surfaced as a visible note at the top of the draft body for the editor reviewer. Validated end- to-end against the live site (draft #123656). - hackernoon, dzone, medium: standard browser flow. Selectors are best-effort and need a one-time validation pass against each live site via --validate-only --headed. medium has no password login, so it relies on a base64-encoded MEDIUM_STORAGE_STATE secret exported from a manually logged-in browser session. Workflow additions: - Detects whether any browser-syndication secret is configured; only installs Playwright + Chromium when something will actually run. - Uploads the Playwright screenshot directory as a CI artifact on any outcome (kept for 14 days), so selector failures are debuggable. - Screenshots dir is gitignored. Per-platform secrets (all optional; missing = platform skipped): FOOJAY_USER, FOOJAY_PASSWORD HACKERNOON_USER, HACKERNOON_PASSWORD DZONE_USER, DZONE_PASSWORD MEDIUM_STORAGE_STATE Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
scripts/website/export_medium_storage.py captures a logged-in Medium session as a base64 blob suitable for the MEDIUM_STORAGE_STATE secret that the browser syndicator's MediumAdapter requires. Three modes: - --from-firefox-profile (no second login): reads cookies.sqlite from the user's existing Firefox profile and builds the storage state JSON directly. Auto-detects the most recently used profile under ~/Library/Application Support/Firefox/Profiles/. Refuses to write state if the profile is not actually logged in (uid cookie missing or prefixed with `lo_`). - --browser firefox|chrome|chromium|msedge: launches Playwright with the requested browser, opens medium.com/m/signin, and polls cookies every 3s until a non-`lo_` uid appears. 10-minute timeout default. - --interactive: same launch but waits on stdin instead of polling (useful when running attached to a real terminal). Output is written as JSON to --output and (unless --no-base64) printed as a base64 blob ready to paste as a repo secret. The local JSON file is gitignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two structural changes to the browser syndicator:
1. DZone — switch from password-based login to a saved Playwright
storageState loaded from a DZONE_STORAGE_STATE secret. DZone's
AngularJS doLogin() requires a reCAPTCHA token (visible in
scope.credentials.recaptchaToken) that Google's invisible reCAPTCHA
does not issue to headless browsers; the auth request is never sent.
Same approach as the existing MediumAdapter.
2. HackerNoon — replace .fill() with .press_sequentially() because the
login inputs are React-controlled. .fill() set DOM .value but never
updated React's internal state, so doLogin() ran with empty fields.
With per-character typing the form actually submits; HackerNoon's
"Invalid email or password" message now surfaces (instead of a
silent no-op) when credentials don't match. Also fail-fast on a
stuck-on-/login URL with the explicit error text.
Helper script renamed export_medium_storage.py ->
export_storage_state.py and generalized to support multiple sites via
--site {medium,dzone}, with per-site cookie host filter and login
detector. Browser-launch path picks Playwright Firefox when --browser
firefox is requested.
Workflow updated for the new DZONE_STORAGE_STATE secret name; gitignore
generalized to exclude all *-storage-state.json scratch files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit only captured the file rename; this one carries the actual code changes for DZone (storage-state auth via DZONE_STORAGE_STATE), HackerNoon (React-friendly press_sequentially typing + fail-fast on stuck-on-/login), the multi-site export helper (--site, profiles for medium and dzone, Firefox cookie host filter), the workflow secret rename (DZONE_USER/PASSWORD -> DZONE_STORAGE_STATE), and the broader *-storage-state.json gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three groups of changes based on review of foojay draft #123656:
1. Body rendering (applies to all syndication targets):
- Strip the trailing Hugo "## Discussion" + giscus shortcode block
so the syndicated copy ends at the actual article body.
- Strip any remaining {{< shortcode >}} forms.
- Replace the markdown-blockquote "What is Codename One?" with an
HTML <aside> styled as a left-bordered callout so it reads as a
sidebar instead of a quote.
2. FoojayAdapter post creation now sets:
- categories=[1722] (Java)
- a `codenameone` tag (created lazily via /wp/v2/tags if missing)
- featured_media: downloads the post's cover image from
www.codenameone.com and uploads it to /wp/v2/media, then assigns
the returned id as the post's featured image
- excerpt from the post's `description` front-matter
- meta._yoast_wpseo_canonical / _title / _metadesc are sent in the
payload as a best-effort; Yoast on foojay does not register these
for REST writes, so they are silently dropped. The canonical URL
is also kept as a hidden HTML comment at the top of the body so
the editor reviewer can paste it into Yoast's metabox.
3. Refactor the WP REST plumbing into _rest_get / _rest_post helpers
and centralise the User-Agent string used by both Playwright and
urllib calls.
Verified against draft #123658: categories, tags, featured_media,
excerpt, sidebar, and footer-strip all confirmed via /wp/v2/posts/...
?context=edit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
REST silently drops _yoast_wpseo_* meta keys (not registered for REST
writes) and Cloudflare blocks both new-post and update form submits to
/wp-admin/post.php with a JS challenge that loses the form payload.
WordPress XML-RPC is unprotected on foojay (Wordfence's app-password
block does not extend to xmlrpc.php), accepts the user's normal
password, and wp.editPost's custom_fields parameter lets us write the
underscore-prefixed Yoast meta keys directly.
After REST creates the draft, the foojay adapter now follows up with
an XML-RPC wp.editPost that sets:
_yoast_wpseo_canonical -> the original codenameone.com/blog/... URL
_yoast_wpseo_title -> the post title
_yoast_wpseo_metadesc -> the post description, trimmed to 155 chars
on a word boundary
Verified end-to-end against draft #123664: Yoast metabox now shows the
canonical, SEO title, and meta description correctly. The visible
canonical HTML comment at the top of the body content is kept as a
secondary signal for the editor reviewer.
Also bumps the cover-image download timeout to 120s after a transient
60s timeout on the prior run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HackerNoon
----------
The /login page form is decorative — submitting it does nothing
useful. The actually-working login lives in a drawer that opens from
the header "Login" button on hackernoon.com. Switched the adapter to:
- navigate to hackernoon.com home page
- dismiss the Iubenda cookie banner
- click header "Login" button to open the drawer
- press_sequentially the email + password into the drawer's React-
controlled inputs (fill() doesn't update React state)
- click drawer "Log In" button
- confirm login by polling for the .hackernoon.com `hasAuthCookie`
Editor flow:
- navigate to hackernoon.com/new
- click "Start Draft" (which routes to app.hackernoon.com/articles/new)
- fill title (textarea[name='title'][placeholder='Title']) and the
SEO description textarea
- paste body into the Quill rich-text editor (div.ql-editor) with
a leading "Originally published at <canonical>" line for the
editorial reviewer
- in normal mode, click "Submit Story for Review!"; in --validate-only
mode, screenshot and exit
dev.to / Hashnode
-----------------
Adds --draft-mode flag to syndicate_blog_posts.py so the API path can
be verified without going live. dev.to switches to published=false;
Hashnode switches from publishPost to createDraft. Production cron
runs without the flag and publishes as before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
foojay ------ - Trim Yoast meta description to 140 chars (was 155 — Yoast's snippet preview cut at ~142 chars, leaving the full version visibly truncated). - dev.to draft URL now points at the dashboard editor instead of the public canonical URL (which 404s for unpublished drafts). storage state helper -------------------- - Auto-detect Firefox cookie expiries that are stored in milliseconds (some cookies in cookies.sqlite use ms despite the documented seconds unit). Anything > ~year 5138 is treated as ms and divided by 1000; fixes "valid expires" Playwright errors when loading state. - Generalize the dzone profile's logged-in detector: matches the Spring Security `remember-me` cookie or any per-session `dz<hash>` cookie (excluding the anonymous `dzuuid`). HackerNoon ---------- - Wait for SPA hydration via networkidle + visible-state poll on the Start Draft button (3s sleep was too short). - Switch from "Submit Story for Review!" (gated on extra fields) to the always-enabled "Save" toolbar button so a draft lands every run. - Disambiguate the "Save" selector with .first since two Save buttons exist on the editor page. - Verified end-to-end: draft created at app.hackernoon.com/articles/<id>. DZone ----- - Use the real editor URL /content/article/post.html (the previously- guessed /articles/new is 404; /articles/create is Cloudflare 403). - Replace untested TinyMCE-iframe selectors with the live Froala editor. Fill title (textarea[name='title']), TL;DR (subtitle), and meta description; set the body via Froala's JS API (FroalaEditor.INSTANCES[0].html.set) instead of clipboard paste, which Froala's paste handler unreliably eats. Medium ------ - Updated to the actual editor — a single contenteditable div.postArticle-content with placeholder "Title\\nTell your story…". Type title, press Enter, paste body, wait for Medium's auto-save to redirect to /p/<draftId>/edit. Medium's editor mounts inconsistently when driven from headless Playwright (Cloudflare bot detection appears to vary by request); behaviour is best-effort and the run may need manual retry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HackerNoon
----------
- Title was being saved with leading characters dropped ("umbing"
instead of "Liquid Glass, Material 3, And A Lot Of Plumbing"). HN's
React-controlled textarea drops chars when fed via press_sequentially
faster than its onChange debounce. .fill() sets the value via the
CDP InputHandler API which React picks up correctly.
- Body was being pasted as raw markdown text and lost all formatting
(no headings, images, links, code blocks). Now converted to HTML
with python-markdown and injected through Quill's clipboard API
(clipboard.dangerouslyPasteHTML), which translates the HTML into
Quill's Delta format. Headings, images, paragraphs, code fences,
and links all render in the editor as a result.
Workflow gets `markdown` added to the pip install for the browser-
syndication step.
State file
----------
Records the existing dev.to + Hashnode syndications of the May 1
liquid-glass post so the daily cron does not create duplicates:
- dev.to: 3620800 -> https://dev.to/codenameone/liquid-glass-...
- hashnode: 69fb2f0263ebe40f84df66db
-> https://debugagent.com/liquid-glass-material-3-...
Production cron runs in publish (non-draft) mode by default so future
posts go live automatically once they hit the 7-day cooling window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Title
-----
.fill() leaves the title empty; press_sequentially with a small
keystroke delay (5-10ms) loses leading characters because HN's React
onChange debouncer drops them. press_sequentially(delay=80) types
slowly enough that every character registers, and the full title
("Liquid Glass, Material 3, And A Lot Of Plumbing") survives instead
of becoming "umbing".
Body
----
Drops the visible "Originally published at <canonical>" line from the
top of the body — the canonical now lives in HN's Story Settings drawer
where it is supposed to be. Description (SEO meta) is also typed via
press_sequentially because the same React-debounce issue applies.
Canonical via Story Settings drawer
-----------------------------------
After saving title + body, the adapter:
- clicks the No button under "Is this story original on HackerNoon?"
(button.css-p9s3bq:text-is('No'), scoped to drawer styling so we
don't accidentally hit a confirm-modal "No" with class="negative")
- fills the canonical URL into the input that appears below
(input.firstSeenAt / placeholder "www.example.com/yourstory")
Cover image
-----------
Downloads the post's header image to a tempfile and uploads it via
the input[type=file][accept*='image'] hidden file input using
Playwright's set_input_files(). No file picker dialog is opened.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sion
HackerNoon drawer
-----------------
The Story Settings drawer has its own internal scroll container, so
Playwright's scroll_into_view_if_needed scrolls the page but leaves
drawer elements outside the viewport — clicks and fills timed out and
none of {description, cover image, canonical} actually persisted.
Switch to JS-driven interactions for everything inside the drawer:
- Description / canonical: set value via React's native value setter
(Object.getOwnPropertyDescriptor(prototype, 'value').set) and
dispatch an `input` event so React's onChange picks it up.
- "Is this story original on HackerNoon?" → No: call .click() on the
button element directly via page.evaluate (no viewport check).
- Cover image: set_input_files on the hidden file input continues to
work (no visibility requirement) — the previous failure was
cascading from the description timeout aborting the run before the
upload completed.
Verified end-to-end: title, description, canonical, and cover image all
persist on the saved draft.
Code blocks
-----------
python-markdown emits <pre><code class="language-X">. Quill's syntax
module reads the language from the <pre>'s data-language attribute, not
the <code>'s class — without it, Quill mis-tags Java as JavaScript.
Added _retag_code_blocks_for_quill() that rewrites those blocks into
<pre class="ql-syntax language-X" data-language="X" spellcheck="false">.
Medium / DZone via Firefox extension
------------------------------------
Both sites are gated by aggressive Cloudflare bot detection that
headless Playwright can't pass reliably. Pivoting to a Firefox
extension that runs inside the user's already-trusted browser session:
scripts/syndication-extension/ -- the extension itself
manifest.json manifest v3
background.js polls a queue file in the repo every 30 min
popup.html/.js pending/completed list + JSON state-patch to paste
back into syndication-state.json
adapters/
common.js shared waitFor / React-setter / file-attach helpers
medium.js title via execCommand insertText, body via
execCommand insertHTML, canonical via Story
Settings panel
dzone.js title/subtitle via React setters, body via
FroalaEditor.INSTANCES[0].html.set, click Save draft
scripts/website/queue_browser_syndication.py
Walks the same eligible-posts logic as the API
syndicator and appends a per-platform task entry
to syndication-queue.json (de-duped by id).
scripts/website/syndication-queue.json
Committed queue file the extension polls. The
daily CI job appends to it; the user's Firefox
processes the queue when online; results land in
syndication-state.json after the user pastes the
patch from the extension's popup.
The Playwright DZoneAdapter and MediumAdapter are removed — they were
unreliable. The browser-syndication script now defaults to
foojay,hackernoon only. A new workflow step runs
queue_browser_syndication.py on every cron run; the commit-state step
now also commits queue updates.
The extension is unsigned, so a temporary install from
about:debugging#/runtime/this-firefox is required and persists until
Firefox is restarted. README inside the extension dir documents
installation, architecture, and how to add new platforms (Bluesky,
Mastodon, etc. — same pattern, different adapter file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HackerNoon charges business sites for canonical-URL support. Without a canonical link back to www.codenameone.com the post becomes a duplicate of the original — which defeats the SEO model the syndication pipeline is designed around. Removed: - HackerNoonAdapter class and all selectors (login drawer, Quill body, Story Settings drawer, cover image, canonical-via-No flow) - HACKERNOON_USER / HACKERNOON_PASSWORD secret passthrough in the workflow + their detection in the browser-creds gate - HN-specific helpers in syndicate_browser_posts.py: _markdown_to_html, _retag_code_blocks_for_quill, _download_to_temp, _escape_html, the unused `re` import Workflow tweak: `markdown` is now installed in its own unconditional step (queue_browser_syndication.py needs it whether or not Playwright runs), so removing HN doesn't break the Medium/DZone queue path. Default platforms for syndicate_browser_posts.py is now `foojay` only. The full active syndication set: API: dev.to, hashnode (syndicate_blog_posts.py) Hybrid: foojay (Playwright login + WP REST + XML-RPC for Yoast) Browser: medium, dzone (Firefox extension via syndication-queue.json) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CodeQL flagged two patterns:
1. Unused local `slug` in publish_to_hashnode() draft branch — dropped.
The draft URL is built from node.id only; the slug was assigned but
never read.
2. Empty except clauses with `pass` in syndicate_browser_posts.py —
added inside-the-body explanatory comments documenting why each
exception is intentionally swallowed:
- _upload_featured_media: media-item title rename is cosmetic;
upload itself already succeeded.
- run_adapter clipboard grant: Firefox/WebKit reject the chromium-
only clipboard-* perms; adapters fall back to editor-specific APIs.
The other empty-except findings were on adapter classes that have
since been removed (HackerNoon, DZone, Medium adapter classes are no
longer in syndicate_browser_posts.py — Medium and DZone moved to the
Firefox extension; HackerNoon was dropped because it charges business
sites for canonical-URL support).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/workflows/blog-syndication.yml) that syndicates new blog posts to dev.to and Hashnode with acanonical_url/originalArticleURLback towww.codenameone.com.scripts/website/syndicate_blog_posts.py(Python stdlib only) picks the oldest post dated after 2026-04-30, at least 7 days old, and not yet syndicated to a given platform; absolutizes relative links/images; and inserts a one-sentence "What is Codename One" blurb right after the fold.scripts/website/syndication-state.jsontracks per-slug, per-platform results so partial failures retry only the failed side. Action commits state updates back tomaster.Setup
Repo secrets required (already added):
DEVTO_API_KEYHASHNODE_TOKENHASHNODE_PUBLICATION_IDThe workflow runs daily at 13:00 UTC and supports
workflow_dispatchwith a dry-run toggle. The first eligible post isliquid-glass-material-3-modern-native-themes(2026-05-01), which becomes a candidate from 2026-05-08 onward.Test plan
python3 scripts/website/syndicate_blog_posts.py --dry-runon 2026-05-06 / 2026-05-07 reports no candidate.--dry-run --today 2026-05-08selects the May 1 post.dry_run=trueafter merge to confirm secrets are wired.🤖 Generated with Claude Code