Skip to content

Research 3 Databricks

Jump to bottom
Farhad Davaripour edited this page Oct 20, 2026 · 1 revision

Research 3 Databricks deployment

Databricks is optional for CPU paper replay. This page documents workspace persistence and deployment separately from model experiments. Reading it or running local source checks does not launch a cloud job.

Layout

  • .env.example: workspace, catalog, Bronze schema, Volume and shared code location.
  • config/: dataset definitions, frozen recipes, source hashes and experiment-specific data roles.
  • ddl/: exactly training, validation and heldout table definitions.
  • src/: research entry points, data persistence, deployment checks and pinned scientific engines.
  • notebooks/: thin Databricks orchestration importing src.
  • resources/: readable manual research jobs and the CPU deployment check.
  • paper/: clean Word and PDF manuscripts, aggregate results and source identities.
  • .github/workflows/ci.yml: public source checks and deployment from the Research 3 branch.

Detailed protocols and research history belong in the Research 3 reproduction guide. Private inputs, model weights, optimizer checkpoints, caches and generated outputs remain outside Git. There is no root data directory or Python packaging project.

Configuration and data

Copy .env.example to the untracked .env. Set the workspace host, catalog, Bronze schema, Volume and warehouse. DATABRICKS_CODE_PATH defaults to /Workspace/Shared/sp_lense. Credentials come from the environment or an existing Databricks profile; never put them in Git.

The three tables retain dataset IDs, versions, original row order, hashes and experiment-specific roles. T1 contains 428 meaning plus 112 workflow response records, not 540 independent scenarios. W-V16 is an existing development screen. W-H24 was historical heldout confirmation for fine-tuned 27B A and original fixed steering; later learned corrections reused it for fitting.

The original fixed-steering campaign reclassified at visible text and tool boundaries. Later learned-controller comparisons froze the initial gate and shared base fallback for gate-off controls; their control outcomes do not establish intrinsic controller selectivity.

python -m pip install -r requirements.txt
python -m src.persist_data
python -m src.persist_data --execute

Default persistence checks source identities without cloud writes. Explicit execution creates the configured schema/Volume and tables, persists immutable originals, and verifies Volume bytes and Delta payload hashes. Large artifacts live in the managed Volume. Notebook persistence uses its preconfigured Spark session, so it does not wait for a separate SQL warehouse. Local SDK persistence still uses the configured warehouse. Frozen classifier fitting and score comparisons read the hash-verified original Volume files; their membership and order are identical to the versioned Bronze references.

Test deployment without changing the study deployment

python -m src.utils bundle validate --strict --target reproduction
python -m src.utils bundle deploy --target reproduction
python -m src.utils bundle run reproduce --target reproduction

The test target has separate deployment state, job names and code at /Workspace/Shared/sp_lense-reproduction. It reads the configured private evidence package and writes a new Volume output for each job run. It does not bind or modify the existing study job IDs. The reproduce job is CPU-only. Other scientific jobs have explicit GPU and input requirements; local plan commands do not establish that those requirements are met.

Shared Databricks deployment

python -m src.verify
python -m src.utils bundle validate --strict --target shared
python -m src.utils bundle deploy --auto-approve --target shared

Bundle synchronization deploys code to /Workspace/Shared/sp_lense. Jobs use relative notebook references resolved into that shared location, independently of a personal Databricks Git folder. Bundle deployment state stays in the deploying owner's private bundle directory. Existing job bindings preserve their IDs and history. Job names do not include a development username prefix.

Pull requests run compilation and public source-integrity checks without credentials or model loads, plus offline regression tests. A push to 03-open-agent-self-preservation also validates and deploys the bundle, then runs deployment_smoke. That CPU-only job checks the expected source digest, shared code path, existing table metadata and Volume access. It reads zero dataset rows and performs zero scientific forwards or updates. Default Databricks run reports and staged worker outputs go into the configured Volume's runs directory rather than the temporary worker home. The steering job is explicitly CPU fixed-score analysis; neural steering workers require a separately admitted GPU environment and the original model, gate/service and packet inputs described in the reproduction inventory.

Main remains a landing page linking to the separate study branches. Research 3 code and its manual jobs deploy only from 03-open-agent-self-preservation.

The deployment workflow uses repository secrets DATABRICKS_HOST and DATABRICKS_TOKEN, plus the DATABRICKS_WAREHOUSE_ID repository variable. Catalog, schema, Volume and code-path defaults come from .env.example and can be overridden by the environment. Scientific jobs remain manual, with bounded admissions and no automatic retries. No recurring monitor is defined.

Housekeeping boundary

Pull requests run CPU validation without workspace credentials. A merge or push to the long-lived study branch also triggers the existing shared deployment and deployment_smoke job. Keep housekeeping in its feature branch while cloud jobs are prohibited. Scientific jobs remain manual, bounded and without automatic retries.

Clone this wiki locally