Self-bootstrapping bundled dev env at the extension root#811
Open
alistair3149 wants to merge 20 commits intomasterfrom
Open
Self-bootstrapping bundled dev env at the extension root#811alistair3149 wants to merge 20 commits intomasterfrom
alistair3149 wants to merge 20 commits intomasterfrom
Conversation
Provides a dev-capable image that does not bake NeoWiki source. NeoWiki is mounted at runtime; composer install runs on first boot. Includes vim/nano/less/make for in-container debugging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The entrypoint runs composer install as PID 1 (root). Without this fix, vendor/ files are root-owned on the host bind-mount and developers need sudo to clean them up. Make the directory www-data-owned and world-writable so cleanup works without elevated privileges. For routine re-installations after first boot, prefer `make composer-install`, which runs as the host UID via `docker compose exec --user` and produces host-owned files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds test_neo (parallel test isolation on bolt 7689 / http 7475), node (TS watcher sidecar against /workspace bind-mount), and mailcatcher (SMTP catcher on MAILCATCHER_PORT) to the dev profile. Documents the v2 local-backend profile placeholder above db. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code review feedback: - node sidecar: restart: unless-stopped causes tight restart loops on npm install failure. Switch to on-failure:3 so transient errors still recover but persistent ones halt. - test_neo: NEO4J_AUTH is hardcoded because NeoWiki's test suite expects a fixed credential. Add a comment so this is not "fixed" later without also updating tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches mediawiki service to locally-built dev-mw target, bind-mounts the NeoWiki source, optionally overlays sister extensions via MW_EXTRA_EXTENSIONS_DIR.
When MW_MODE=dev, enables exception details, debug SQL, disables caches, and points SMTP at mailcatcher. Defaults preserve production behavior. Adds LocalSettings.local.php opt-in extension point for per-worktree overrides (e.g., loading sister extensions).
MW_MODE for dev/production switch, MAILCATCHER_PORT, MW_EXTRA_EXTENSIONS_DIR for sister-extension overlay, SHARED_BACKEND placeholder for v2.
…ults Adds make dev, bash, phpunit, cs, tsci, composer-install, ts-install, ts-build, reset, update-dot-php. Derives COMPOSE_PROJECT_NAME from parent directory for collision-free worktree parallelism. Auto-allocates MW_SERVER_PORT on first run. Health-gates make dev on /Special:Version.
Closes the public contributor onboarding gap for issue #120. Documents make dev, test/lint targets, per-worktree behavior, and LocalSettings.local.php opt-in. Also gitignores Docker/LocalSettings.local.php so the per-worktree opt-in file stays untracked.
Public contributors and PW devs now run 'make dev', 'make phpunit', 'make tsci' etc. directly from mediawiki/extensions/NeoWiki/. The Docker/ directory keeps the stack files (Dockerfile, compose, scripts) but no longer needs its own Makefile. PHP test targets (phpunit/cs/stan/psalm) are dual-mode: when invoked from outside the container they exec via docker compose; when invoked from inside the container (e.g. by the existing AGENTS.md docker compose exec wrapper) they run the binaries directly. TS targets exec into the running 'node' sidecar. Project name is lowercased to satisfy compose v2's naming rule. The set-port.sh script honours an ENV_FILE env var so it can write to Docker/.env from the extension root.
The dev-mw Dockerfile stage now bakes SettingsTemplate.php as LocalSettings.php (matching final-mw) so install-db's swap dance has something to swap with on first boot. The _wait-mw health gate now waits for any HTTP response rather than grepping for 'NeoWiki|Jeroen', because before install-db runs the wiki redirects to the setup page. Drops ts-install / ts-build from _first-run-seed: the node sidecar already runs npm install && npm run build:watch on startup, so making the seed call them in parallel would race on file locks.
Port 1080 is commonly reserved on Windows hosts (SOCKS proxy) and Docker Desktop on WSL2 fails to bind to it. 1180 is unreserved.
PHPUnit (and other PHP tools to a lesser degree) hangs at startup when invoked via 'docker compose exec -T' because that mode does not fully close stdin. The MW test bootstrap blocks reading from it. Adding '< /dev/null' to the inside-container invocations forces a closed stdin. Also relaxes the _wait-mw probe to use 'curl -sS' instead of '-fsSL', so the gate accepts 5xx responses (expected before install-db has created the wiki tables on first boot).
Each worktree's mailcatcher used to bind the default 1180 host port, which collides as soon as a second worktree starts. Derive the mailcatcher port from MW_SERVER_PORT (offset from 8484/1180), so each worktree gets its own MW + mailcatcher pair without a separate allocation pass.
- Add a bootstrap make target that clones MW core into Docker/mediawiki/ (the gitignored build prerequisite) and seeds Docker/LocalSettings.local.php on the host. make dev depends on it, so a fresh clone is now self-bootstrapping. - Bind-mount Docker/LocalSettings.local.php into the container instead of expecting the entrypoint to materialize it inside. The override mechanism documented in the README now actually works. - Replace the MW_EXTRA_EXTENSIONS_DIR /dev/null fallback with a tracked empty directory (Docker/empty-extensions-extra/), so the bind mount is always valid even on Compose versions that reject device files as bind sources. - Stop tracking Docker/.env: it ships as Docker/.env.dist and the Makefile copies it on first run. set-port.sh's runtime port writes no longer show up in git status. - Drop the RANGE_START carve-out in set-port.sh; trust is_port_free to handle 8484 like any other port. - Apply '< /dev/null' to the outer docker-compose-exec wrapper for all dual- mode targets so the docker-exec stdin hang fix matches the documented mitigation. - Use the lightweight 'docker --version | grep podman' check instead of 'docker info', which is slow and can hang briefly when the daemon is busy. - Add _wait-node prerequisite to ts-* targets so the first invocation after make dev does not race the node sidecar's initial 'npm install'. - Flag the dev-mw image as DEV-ONLY in the Dockerfile and document the build-time composer-merge-plugin caveat (NeoWiki-declared root-level deps would not get merged at build time). - Trim dead code from dev-entrypoint.sh now that LocalSettings.local.php always exists via the host bind mount. - Generalize 'PW internal devs' wording in compose/.env.dist/Makefile to vendor-neutral language (this is a public repo). - README example uses a generic 'SomeExtension' instead of a vendor-specific extension name.
…rst start The build-time composer install in the Dockerfile happens before the NeoWiki bind-mount exists, so composer-merge-plugin cannot pull extensions/NeoWiki/composer.json into the root vendor. On a fresh-clone make dev, MediaWiki's autoloader then could not find NeoWiki's runtime dependencies (jeroen/file-fetcher, opis/json-schema, etc.) and demo-data import failed with 'Class FileFetcher\\SimpleFileFetcher not found'. The entrypoint now runs composer update at MW root once on first start (idempotent via a marker file in vendor/) so the lock regenerates with the merged include list and NeoWiki's deps land in the root vendor. Discovered while smoke-testing the bootstrap path against a wiped state (no Docker/mediawiki, no .env, no LocalSettings.local.php, no volumes, no image). After this fix the full make dev sequence succeeds end-to-end.
- set-port.sh: allocate MW_SERVER_PORT (8484-8499) and MAILCATCHER_PORT (8025-8040) from independent ranges. Previously mailcatcher was derived from the MW port, so an explicit `port=9000` pushed mailcatcher to 1696. Each port is now is_port_free-checked under a single lock taken once for the whole pass. - Default MAILCATCHER_PORT bumped from 1180 to 8025, matching the MailHog/mailpit convention used elsewhere in the dev tooling ecosystem. 1180 was a workaround for 1080 (SOCKS) being commonly bound on Windows/WSL2; 8025 sits in the 8xxx dev-tooling cluster. - Add an opt-in Docker/docker-compose.tools.yml overlay that exposes Neo4j Browser (7474) and Bolt (7687) to the host. Reachable via `make dev-tools` from the extension root or `make dev-tools` from the worktree-wrapper Makefile. Single-worktree by default; override NEO_BROWSER_PORT / NEO_BOLT_PORT for parallel tools-mode worktrees. - Document the reserved host port ranges in Docker/README.md and link to it from the extension README and AGENTS.md. - Replace the hardcoded `http://localhost:8484` references in the extension README with explicit notes that the port is auto-allocated.
fs_overlay/usr/local/etc/php/conf.d/opcache-recommended.ini sets opcache.validate_timestamps=0 (production tuning) and is shared by every image stage. In dev that means PHP edits to bind-mounted files are never picked up without a container restart, including LocalSettings.local.php overrides — the documented per-worktree extension point. A dev-only conf.d drop-in (zz-dev-opcache.ini) flips the flag back on for the dev-mw stage. Verified empirically: with the fix, edits to LocalSettings.local.php propagate within revalidate_freq (~2s); without the fix, the same edit was still invisible after 30s. Also clarifies the docker-compose.dev.yml comment: LocalSettings.local.php is what users edit; SettingsTemplate.php stays baked.
Member
|
Interesting approach Seems like a mediawiki clone ends up being on the host fs, which is great for IDE autocomplete and AI grep. However composer dependencies added to |
Member
|
PR LGTM, though I have not tried it yet.
Goal: a fully-populated MW Today the relationship between host and container is one-way for MW core: This matters for IDE autocomplete and for AI tooling that reads what's on disk. Both work much better when host vendor matches what the container actually runs. Possible fix, kept consistent with the self-bootstrapping design:
Both are small and orthogonal to the rest of the Makefile. Happy to do as a follow-up rather than block this PR. |
Host's Docker/mediawiki/vendor/ used to be whatever MW's REL1_43 git provided (the WMF deployment submodule, production deps only). The container's composer install/update wrote into image layers, never visible to the host. IDE autocomplete, AI grep, and any host-side static analysis saw a partial vendor that did not match what MW was actually loading. Bind-mount /var/www/html/w/vendor onto the host at Docker/mediawiki/vendor/ so the dev-entrypoint's composer update (with composer-merge-plugin pulling in NeoWiki's deps) writes through. After first boot the host sees the full post-merge vendor — including jeroen/file-fetcher, laudis/neo4j-php-client, opis/json-schema, and the dev tooling. Drop the now-redundant 'composer install' from the dev-mw Dockerfile stage: the bind mount shadows that vendor anyway, and the entrypoint already runs composer update on first boot. Saves ~30s of image build. Entrypoint chowns the bind-mounted vendor to www-data and chmods ugo+rwX so the host user can clean up despite cross-namespace UIDs, matching the existing NeoWiki-vendor handling. Verified: 52 packages on host vendor after first boot, including laudis/neo4j-php-client and jeroen/file-fetcher (the merge-plugin contributions). Marker file at vendor/.neowiki-merged correctly suppresses re-running composer update on subsequent boots. Discussed in #811 (comment)
Member
Author
Done in 32decc0: bind-mount Verified after a clean first boot: 52 packages on host vendor, including |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Converts
Docker/from a try-it-out-only stack into the canonical NeoWikidev environment, with a single
makeentry point at the extension root.A fresh clone now goes from zero to a running wiki with
make dev.Summary
dev-mwDockerfile stage anddocker-compose.dev.ymloverlay with abind-mounted NeoWiki source,
MW_MODE=dev-awareSettingsTemplate.php,and an opt-in
LocalSettings.local.phphook.make dev,make phpunit,make cs,make tsci,make bash,make reset, etc. live at the extension root in a single dual-modeMakefile (proxies via
docker compose execfrom the host; runs binariesdirectly inside the container).
make bootstrapclones MW core into the gitignoredDocker/mediawiki/,so a public clone is self-bootstrapping.
MW_SERVER_PORT8484-8499,
MAILCATCHER_PORT8025-8040.make dev-toolsopt-in addsNeo4j Browser/Bolt at 7474/7687.
Docker/.envnow ships asDocker/.env.distand is auto-copied onfirst run; runtime port allocation no longer dirties the working tree.
opcache.validate_timestamps=1re-enabled in the dev image so PHPedits (including
LocalSettings.local.phpoverrides) land on the nextrequest without a container restart.
Test plan
make devsucceeds end-to-end andhttp://localhost:8484/index.php/Main_Pagerenders.make phpunit filter=SchemaNameTest,make phpcs,make ts-lintpass.
make dev-toolsexposes Neo4j Browser athttp://localhost:7474.port=8488correctly allocatesMW_SERVER_PORT=8488whilekeeping
MAILCATCHER_PORTin its 8025-8040 range.Docker/LocalSettings.local.phppropagate to the runningcontainer within ~2s, no restart needed.
make stop/make down/make resetwork.