The open web, forged into datasets. DataForge is a Zalize product.
Open-source data engineering pipelines behind DataForge — a data-asset factory that collects public web data only (no logins, no paywalls, no CAPTCHA bypass), cleans it, and packages it into analysis-ready datasets: 21 data lines, 150M+ records in production.
This repo contains the collection / cleaning / packaging code, data dictionaries, and free 500-row samples for every data line — plus DataForge Open Data: 21 data lines (45 packages) whose full datasets are free to download, no signup. Everything else is available on the DataForge portal. What is open vs. closed is documented in the selective open-source policy.
These lines are built from sources whose official license permits free redistribution
(US government public-domain data, CC BY 4.0, UK Open Government Licence v3.0), so we publish the
complete packages — the same ones sold on the portal — for free. The table below shows the
first wave; the full 21-line / 45-package list is in the machine-readable index
(https://dl.zalize.com/open-data) and on the open-data portal:
| Dataset | Free full downloads | Rows | License |
|---|---|---|---|
| US work-visa & salary disclosures (H-1B, PERM, LCA) | S · M · L | 14.5M | Public domain (US DOL/USCIS, 17 U.S.C. §105) |
| US public-company financials, insider trades & 13F (SEC EDGAR) | S · M · L | — | Public domain (SEC EDGAR) |
| SEC Form D private placements & fundraising events | S · M · L | 768K | Public domain (SEC) |
| Clinical trials & drug approvals (ClinicalTrials.gov, FDA) | S · M | 1.06M | ClinicalTrials.gov terms (free reuse) |
| US patents, citations & assignee graph (USPTO/PatentsView) | S · M | 255M citations | CC BY 4.0 (PatentsView) |
| Government tenders & contract awards — US federal slice (SAM.gov) | S | — | Public domain (SAM.gov) |
| Business POI & food-hygiene locations (UK FHRS + US city open data) | S · M · L · parquet · csv · jsonl | — | OGL v3.0 + US city open-data terms |
- Machine-readable index (ids, sizes, sha256, licenses):
https://dl.zalize.com/open-data - Field dictionaries & datasheets: under
datasets/<line>/and on each portal page. - Downloads support HTTP Range resume (
curl -C -,aria2c); responses carryx-content-sha256. Rate-limited per IP/day to keep it free for everyone.
Citation — free to use, share and build on (including commercially). When you publish or redistribute, please credit DataForge (data.zalize.com) with a link back to https://data.zalize.com, and keep upstream credits for attribution licenses (PatentsView CC BY 4.0; “Contains public sector information licensed under the Open Government Licence v3.0”):
DataForge Open Data, https://data.zalize.com — US patents package built from USPTO/PatentsView (CC BY 4.0).
Why aren't the other lines free? Their upstream sources don't grant redistribution rights (scraped job postings, app-store reviews, real-estate portals…) or the packages are value-added derived products — see the open-data program audit and the selective open-source policy.
| Pipeline | Dataset | Records (latest run) | Free sample | Full dataset |
|---|---|---|---|---|
pipelines/jobs |
Tech job postings with parsed salaries (Greenhouse/Lever/Ashby/Workable/SmartRecruiters) | 394,300 | sample | portal |
pipelines/visa_salaries |
US work-visa & salary disclosures (H-1B, PERM, LCA) | 14,535,607 | samples | portal |
pipelines/appstore |
App Store & Google Play apps + review-derived sentiment metrics | 23,435 apps | samples | portal |
pipelines/appstore_opportunity |
Paid-validated app niche index / category opportunity scans | 2,101 niches | — | portal |
pipelines/shopify |
Shopify DTC product catalogs + weekly price history | 533,479 SKUs | samples | portal |
pipelines/clinicaltrials |
Clinical trials & drug approvals (ClinicalTrials.gov, CTIS, ISRCTN, FDA) | 1,058,262 | samples | portal |
pipelines/sec |
US public-company financials, insider trades & 13F holdings (SEC EDGAR) | — | — | portal |
pipelines/tenders |
Government tenders & contract awards (US federal + global portals) | 5,956,897 notices | samples | portal |
pipelines/macro |
Global macro & trade indicators, unified country × indicator × year long table (WDI/OECD/Eurostat + IMF-WEO/Comtrade derived) | 9,072,327 observations | sample | portal |
pipelines/patents |
US patents, citations & assignee graph (USPTO / PatentsView) | 255,347,795 citations | sample | portal |
pipelines/global_patents |
Global patent publications metadata: EP / WO / CN / JP / KR (EPO DOCDB) unified with US grants | 105,322,954 publications | sample | portal |
pipelines/papers |
Open-access research papers, authorships & citations (OpenAlex, PMC OA) | 47,277,940 | sample | portal |
pipelines/podcasts |
Podcast shows, episodes & charts | 30,071,597 episodes | samples | portal |
pipelines/steam |
Steam games, review metrics, prices & player counts | 245,213 | samples | portal |
pipelines/techstack |
Website technology-stack detection | 495,363 domains | sample | portal |
pipelines/youtube |
YouTube channels & videos metadata | 6,056,231 | samples | portal |
pipelines/medicare_providers |
US Medicare providers: physician master + prescribing + pharma payments (CMS) | 9,671,889 | samples | portal |
pipelines/nonprofits_990 |
US nonprofits master file + IRS 990 financials & executive compensation | 5,807,612 | samples | portal |
pipelines/fda_regulatory |
FDA regulatory intelligence: recalls, warning letters, MAUDE device events | 25,368,161 | samples | portal |
pipelines/courtlistener |
US court opinions, dockets & judges metadata (CourtListener bulk) | 71.7M dockets / 10.1M clusters | sample | portal |
| methodology | Hiring signals: company tech adoption, team building, geo expansion (closed-source derived product) | 115,976 | — | portal |
| methodology | Company360 cross-source company master (SEC × patents × gov-contracts × jobs × clinical; closed-source derived product) | 1.3M+ source entities | — | portal |
pipelines/realestate |
US residential properties & sale events (government records) | 7,009,514 events | samples | portal |
pipelines/gleif |
Global company registry & corporate ownership graph (GLEIF LEI, CC0) | 3,393,673 entities / 483,370 ownership edges | sample | portal |
pipelines/saasreviews |
B2B software products, reviews & pricing plans | — | samples | portal |
pipelines/formd |
SEC Form D private placements & fundraising events (official structured data sets, 2008Q1+) | 768,749 events | sample | portal |
Record counts come from each line's latest production run manifest; see each dataset's
DATASHEET.md / data dictionary under datasets/.
pipelines/<line>/— collection, cleaning, QC, and packaging code for each data line (Python).pipelines/common/— shared crawling-resilience infrastructure: adaptive rate limiting, proxy pool, session profiles, checkpointed resumable state, run metrics. Seepipelines/common/README.md.datasets/<line>/— data dictionaries, datasheets, bundle (slice) definitions, and free 500-row samples.datasets/build_bundle.py— generic S/M/L bundle builder that turns abundle.jsonslice definition into distributable zip packages.docs/— the selective open-source policy and methodology docs for closed-source derived products.
Documented per line in each DATASHEET.md:
- Public data only — no login-walled or paywalled content, no CAPTCHA solving, no forged authentication.
- Polite crawling — per-host adaptive rate limiting, identified user agent, robots.txt awareness.
- No PII harvesting — person-level identifiers are excluded or pseudonymized.
- Copyrighted expressive content (review text, transcripts, JD full text) is never redistributed openly — public packages carry derived metrics only.
- Provenance — every record carries source URL / identifier and collection timestamp.
git clone https://github.com/wookat/dataforge-pipelines.git
cd dataforge-pipelines
python -m venv .venv && source .venv/bin/activate
pip install -r pipelines/jobs/requirements.txt # per-line requirements
# run the jobs pipeline (discover boards → fetch → normalize → package)
python -m pipelines.jobs.run --helpEach pipeline has its own README with line-specific run instructions.
- Code: Apache-2.0.
- Sample data (files under
datasets/*/sample*/): free to use for evaluation and research with attribution ("Data: DataForge — data.zalize.com"). Full datasets are licensed separately via the portal.
When you use DataForge data in research or publications, replace <slug> / <Dataset Name>
with the dataset's portal slug and title:
@misc{dataforge_<slug>,
author = {{DataForge}},
title = {<Dataset Name>},
year = {2026},
publisher = {DataForge (Zalize)},
howpublished = {\url{https://data.zalize.com/datasets/<slug>}},
note = {Collected from publicly accessible sources; see datasheet for upstream attribution}
}Full citation guidelines: https://data.zalize.com/citation.
data.zalize.com · Open Data · Hugging Face · Kaggle · Apify · Contact: data@zalize.com
PRs welcome — new source connectors, bug fixes, schema improvements. See CONTRIBUTING.md.