RAM-AutoResearch

简体中文 · Roadmap · Naming · Experiments · Domain Directions

AutoResearch platform for recommendation, advertising, and marketing value optimization.

RAM-AutoResearch is an open-source experimental framework for testing model structure, feature engineering, feature interaction networks, serving latency, and business metrics across recommendation, advertising, and marketing systems. The Python package namespace is ram_autoresearch.

RAM stands for:

• Recommendation: retrieval, ranking, reranking, feature interaction, and model structure iteration.
• Advertising: ads delivery, RTA, RTB, OCPC/OCPM, auction simulation, and value-based bidding.
• Marketing: uplift modeling, treatment policy evaluation, LTV prediction, and long-term value optimization.

AutoResearch means the project is not only a model zoo. It is designed to turn papers, local implementations, datasets, profiles, and experiment reports into a repeatable research loop.

Why This Project

Production recommendation and advertising systems rarely improve from model architecture alone. A useful research loop also needs feature-domain analysis, ranking and bidding metrics, segment-level business interpretation, model latency profiling, and repeatable experiment reports. RAM-AutoResearch is built around that loop.

Core principles:

• Use native PyTorch and TensorFlow APIs for model structures, without wrapping third-party recommender model libraries.
• Keep local smoke experiments small by default, usually MB-scale datasets or deterministic synthetic fallbacks, while keeping the design compatible with GB-scale datasets and remote training.
• Treat retrieval, ranking, reranking, ads delivery, uplift, LTV, and bidding as connected but separate research layers.
• Store every experiment with metrics, feature manifests, model structure, latency summaries, operator-level profiles, and HTML/Markdown reports.
• Maintain a lightweight research tree so hypotheses, run evidence, outcomes, and next steps survive beyond a single experiment report.
• Provide a validator-gated AutoResearch loop engineering path: evidence, proposal, plan, safe execution, judgement, research-tree update, and next hypotheses.

Current Coverage

Area	Implemented	Notes
Retrieval	Recommendation closed-loop runner, MovieLens two-tower, DSSM, side-feature DSSM, graph recall, supply/user cold-start hooks, multi-vector/session recall, Kuai-style gap-closure validation, position/popularity debiasing and DR ranking value	In-batch negatives, hard negatives, popularity-debiased weights, brute-force ANN hooks, optional FAISS/ScaNN contract
Ranking	MLP, DCN, Wide & Deep, DeepFM, PNN, AFM, xDeepFM, AutoInt, FiBiNET, BST, SIM, DSIN, DIN, DIEN-style GRU attention, ESMM, MMoE, PLE, feature ablation, feature-domain interaction search, hard-sample diagnostics	PyTorch native implementations plus TensorFlow graph/session variants for DCN and DIN
Reranking	MMR, DPP-style diversity, constrained greedy reranking, listwise greedy, learned-weight listwise sweep, neural ListMLE reranker, value-aware slate scoring, slate latency sweeps	Slate relevance/value/diversity/freshness/exposure summaries
Advertising	Real Criteo ads sample closed loop, optional Avazu/Ali-CCP/iPinYou log entry points, OCPC, OCPM, RTB, RTA, calibration, learned win-rate proxy, delayed conversion, bid landscape, floor sensitivity, IVT, creative optimization, campaign budget allocation, campaign health, cross-campaign budget shift, frequency caps, attribution windows, reserve sweeps, counterfactual policy value	Real sample validation plus deterministic fallback streams when optional public logs are absent
Marketing	Real Criteo marketing sample closed loop, optional CRM/order/channel/coupon entry point, uplift modeling, X/R/DR uplift helpers, causal-forest proxy, budget/channel constrained policy, coupon/contact optimization, churn plus value, uncertainty LTV, LTV-to-bidding, randomized holdout, CUPED, switchback, geo lift, synthetic control, lifecycle diagnostics	Criteo sample lacks true monetary/channel/coupon fields, so those are deterministic policy proxies until a CRM/order dataset is cached
Frameworks	PyTorch and TensorFlow	TensorFlow uses TF2 packages with TF1-compatible graph/session style; DCN, DIN, LTV, ads value/win-rate, neural listwise reranking, and debiasing ops are covered
Profiling	Torch and TensorFlow inference profiles, source hotspot mapping, latency budgets, regression gates, TorchScript export benchmark	Trace JSON artifacts can be opened in browser tracing tools
Reporting	Single-experiment and comparison reports	JSON, Markdown, and static HTML outputs
Datasets	MovieLens, Criteo tiny/local, uplift, ads synthetic, Online Retail LTV, Avazu/Ali-CCP/KuaiRec/KuaiRand/Tenrec/iPinYou/CRM metadata and schema normalizers, download/checksum helpers, deterministic splits	MB-scale defaults; larger optional datasets are registered separately
Paper Ingestion	arXiv API parser, local text/PDF folder ingestion skeleton, paper-to-model-change proposal runner	PDF ingestion uses optional pypdf
Public Platform	Built-in provider registry and CLI listing for datasets, models, metrics, reports, and remote training providers	First plugin contract is local and dependency-free
Deployment Research	TorchScript export, optional ONNX export, batch-size sweep, CPU/CUDA/MPS parity check, Chrome trace export, benchmark dashboard, source hotspot mapping	Local Mac CPU validation; GPU rows are reported when available

Open-Source Readiness

The repository includes the public project files expected for a GitHub release:

• LICENSE: MIT license.
• CONTRIBUTING.md: contribution rules for datasets, models, experiments, and reports.
• CODE_OF_CONDUCT.md: community behavior standard.
• SECURITY.md: supported security-reporting channel and scope.
• CHANGELOG.md: public release history.
• CITATION.cff: citation metadata for research users.
• OPEN_SOURCE_CHECKLIST.md: remaining release tasks.
• docs/DOMAIN_DIRECTIONS.md: domain-specific research directions and open gaps.
• docs/ROADMAP.md: status-labeled roadmap.
• docs/EXPERIMENTS.md: runnable experiment entry points.
• docs/TESTING.md: local and CI testing policy.
• docs/RELEASE_HYGIENE.md: files that must stay out of a public release.
• .github/workflows/ci.yml: CPU test workflow for supported Python versions.
• docs/BRANDING.md: logo and square project icon assets.

Model Library

Native model implementations currently include:

• Retrieval: two-tower user/item encoder, feature-enhanced user tower, metadata-only item tower for supply cold-start, DSSM, side-feature DSSM, graph recall, sampled softmax, hard negatives, brute-force ANN evaluation hooks, and optional FAISS adapter.
• Ranking: MLP, DCN cross network, DCN hard-example replay, Wide & Deep, DeepFM, PNN, AFM, xDeepFM, AutoInt, FiBiNET, BST, SIM, DSIN, DIN pooling, DIN attention, DIEN-style GRU interest evolution with attention, ESMM, MMoE, and PLE.
• Reranking: MMR, DPP-style diversity, constrained greedy reranking, and value-aware slate scoring, plus learned-weight listwise objective sweeps.
• Ads: pCTR/pCVR/value prediction heads, RTA accept model, bidding utilities for CPC, OCPC, OCPM, RTB, blended short-term plus LTV value, and ad lifecycle segment diagnostics, calibration, delayed conversion correction, bid landscape, native PyTorch win-rate model and learned win-rate proxy, floor sensitivity, invalid-traffic diagnostics, creative optimization, campaign budget allocation, and IPS/SNIPS/DR counterfactual evaluation.
• Marketing: S-learner, T-learner, TARNet-style uplift model, LTV MLP regressor, zero-inflated lognormal LTV model, X/R/DR uplift helpers, survival LTV, BG/NBD-style repeat purchase estimates, Gamma-Gamma-style value estimates, treatment-cost incremental profit curves, channel policy, coupon policy, causal-forest proxy, constrained contact optimization, churn plus value, uncertainty LTV, LTV-to-bidding, next-best-action, and lifecycle segmentation.
• TensorFlow graph/session coverage: DCN, DIN pooling/attention, LTV MLP, ZILN LTV, ads CTR/CVR/value heads, win-rate model, neural listwise reranker with ListMLE, position/popularity debias weights, weighted logloss, and doubly robust ranking value.

Experiment Artifacts

Each experiment run is written under experiments/runs/<run_id>/ with a common layout:

• artifacts/metrics.json: offline, business, and latency metrics.
• artifacts/feature_manifest.json: selected fields and feature encoding.
• artifacts/model_structure.json: model inputs, towers, crosses, losses, and architectural changes.
• artifacts/operator_profile.json: summarized operator-level inference cost.
• artifacts/profiles/*.json: browser-readable trace files.
• artifacts/research_node.json: optional hypothesis-tree context for the run.
• artifacts/events.jsonl: optional evidence events collected during or after the run.
• reports/experiment_report.html: static experiment report.
• reports/experiment_report.md: Markdown report.

Quick Start

python -m pip install -e ".[dev]"
python -m ram_autoresearch.cli datasets list
python -m ram_autoresearch.cli datasets select --max-size-mb 1024 --task recommendation --task marketing
python -m ram_autoresearch.cli datasets download-public --dry-run --max-size-mb 10240 --output data/raw/public_download_plan.json
python -m ram_autoresearch.cli plugins list
pytest -q

Run a minimal end-to-end experiment:

python -m ram_autoresearch.training.criteo_dcn_experiment

The command writes a run directory under experiments/runs/. Open the generated HTML report:

ram-ar reports build experiments/runs/<run_id>

Profile traces are written under artifacts/profiles/*.json and can be opened with browser tracing tools such as chrome://tracing or Perfetto UI.

Run representative experiments:

python -m ram_autoresearch.training.movielens_two_tower_experiment
python -m ram_autoresearch.training.recommendation_closed_loop_experiment
python -m ram_autoresearch.training.movielens_real_closed_loop_experiment
python -m ram_autoresearch.training.recommendation_gap_closure_experiment
python -m ram_autoresearch.training.recommendation_debiasing_experiment
python -m ram_autoresearch.training.graph_retrieval_experiment
python -m ram_autoresearch.training.realworld_validation_experiment --max-source-mb 10240
python -m ram_autoresearch.training.criteo_dcn_experiment
python -m ram_autoresearch.training.criteo_din_experiment
python -m ram_autoresearch.training.ranking_zoo_experiment
python -m ram_autoresearch.training.sequence_ranking_zoo_experiment
python -m ram_autoresearch.training.reranking_experiment
python -m ram_autoresearch.training.neural_reranking_experiment
python -m ram_autoresearch.training.hillstrom_uplift_experiment
python -m ram_autoresearch.training.marketing_real_closed_loop_experiment
python -m ram_autoresearch.training.marketing_gap_closure_experiment
python -m ram_autoresearch.training.marketing_experiment_design_experiment
python -m ram_autoresearch.training.online_retail_ltv_experiment
python -m ram_autoresearch.training.ads_real_closed_loop_experiment
python -m ram_autoresearch.training.ads_gap_closure_experiment
python -m ram_autoresearch.training.ads_production_strategy_experiment
python -m ram_autoresearch.training.production_model_deepening_experiment
python -m ram_autoresearch.training.performance_deployment_experiment
python -m ram_autoresearch.training.realworld_validation_experiment
python -m ram_autoresearch.training.realworld_validation_experiment --max-source-mb 1024
python -m ram_autoresearch.training.budgeted_readiness_experiment --max-size-mb 1024 --max-rows 10000
python -m ram_autoresearch.training.tensorflow_gap_coverage_experiment
python -m ram_autoresearch.training.value_based_bidding_experiment
python -m ram_autoresearch.training.autoresearch_proposal_experiment
python -m ram_autoresearch.training.autoresearch_loop_experiment
python -m ram_autoresearch.training.benchmark_dashboard_experiment

Run a local <=1GB completion pass:

python -m ram_autoresearch.training.budgeted_readiness_experiment --output-root experiments --max-size-mb 1024 --max-rows 10000
python -m ram_autoresearch.training.movielens_real_closed_loop_experiment --output-root experiments
python -m ram_autoresearch.training.criteo_dcn_experiment
python -m ram_autoresearch.training.criteo_uplift_experiment --output-root experiments --data-path data/raw/uplift/criteo/criteo-uplift-v2.1.sample.csv.gz --max-rows 5000
python -m ram_autoresearch.training.online_retail_ltv_experiment
python -m ram_autoresearch.training.ads_real_closed_loop_experiment --output-root experiments --data-path data/raw/uplift/criteo/criteo-uplift-v2.1.sample.csv.gz

Build or compare reports:

ram-ar reports build experiments/runs/<run_id>
ram-ar reports compare experiments/runs/<run_a> experiments/runs/<run_b> --output experiments/reports/comparison.html

Track experiment hypotheses:

ram-ar research init experiments/research_tree.json --objective "Improve Criteo ranking AUC without increasing p95 latency"
ram-ar research add experiments/research_tree.json --id dcn-hard-example --hypothesis "Hard-example replay improves sparse tail ranking" --domain ranking --dataset criteo
ram-ar research update experiments/research_tree.json dcn-hard-example --status accepted --run-dir experiments/runs/<run_id> --metric auc=0.72 --score 0.72 --insight "Tail ranking improved on the tiny profile"
ram-ar research view experiments/research_tree.json

Run the AutoResearch loop:

ram-ar autoresearch loop --business-note "Rerank slate with neural listwise diversity objective."

Documentation

• Experiment guide
• Testing guide
• Release hygiene
• Domain directions
• Dataset cards: MovieLens, Criteo, Criteo Uplift, Hillstrom-style Uplift, Online Retail LTV, Public datasets under 10GB, Synthetic Ads Delivery

Roadmap

The full roadmap is tracked in docs/ROADMAP.md with explicit status labels: ☑️ Done, 🔄 In Progress, ☐ Not Started, and ⏳ Later.

Track	Status
Open-source baseline	☑️ Done
Naming and namespace	☑️ Done
Dataset registry	🔄 In Progress
Retrieval	🔄 In Progress
Ranking	🔄 In Progress
Reranking	🔄 In Progress
Advertising	🔄 In Progress
Marketing	🔄 In Progress
Reporting and profiling	🔄 In Progress
AutoResearch automation	🔄 In Progress
Public platformization	☑️ Done
TensorFlow parity	🔄 In Progress
External plugin ecosystem	☐ Not Started
LLM extension	⏳ Later

Project Status

This project is in early research-platform form. The current implementation is best treated as a reproducible local laboratory for model and feature iteration, not a production serving system. Contributions should keep experiments small, deterministic, well tested, and explicit about which claims are backed by local metrics, papers, or open-source implementation audits.