Skip to content
@opendatalab

OpenDataLab

OpenDataLab provides access to numerous significant open-source datasets.

English🌎|简体中文🀄

Note

📚 In 2025, we have open-sourced a high-quality multilingual dataset, WanJuan 3.0 (WanJuan Silu)

🧾 ​​January 2025: Initial Release of Multilingual Pre-training Corpus​​: Primarily text-based data.Collected publicly available web content, literature, patents, and more from 5 countries/regions.Total data size exceeds ​​1.2TB​​, with ​​300 billion tokens​​, achieving international leadership.The initial release includes ​​Thai, Russian, Arabic, Korean, and Vietnamese​​ sub-corpora, each exceeding ​​150GB​​.Leveraging the ​​"InternLM" Intelligent Tagging System​​, the research team categorized each sub-corpus into ​​7 major classes​​ (e.g., history, politics, culture, real estate, shopping, weather, dining, encyclopedias, professional knowledge) and ​​32 sub-classes​​, ensuring localized linguistic and cultural relevance.Designed for researchers to easily retrieve data for diverse needs.
​​Download Links​​: RussianArabicKoreanVietnameseThai.


🌏 ​​March 2025: Second Release of Multilingual Multimodal Corpus​​: which comprises over 1.2TB of indigenous textual corpora from five countries. Each subset includes seven major categories and 34 subcategories, covering a wide range of local characteristics, such as history, politics, culture, real estate, shopping, weather, dining, encyclopedic knowledge, and professional expertise. Here are the download links for the five subsets, and we welcome everyone to download and use them.

Comprises ​​4 data types​​:

  • Image-Text​​: Over ​​2 million images​​ (raw size: 362.174GB).
  • Audio-Text​​: ​​200 hours​​ of ultra-high-precision annotated audio per language.
  • Video-Text​​: Over ​​8 million video clips​​ (raw duration: 28,000+ hours; refined to 16,000+ hours of high-quality content).
  • Localized SFT (Supervised Fine-Tuning)​​:184,000 SFT entries​​ covering local culture, daily conversations, code, mathematics, and science.​​23,000 entries per language​​, including ​​3,000 culturally unique Q&A pairs designed by local residents​​ and ​​20,000 translated entries​​ filtered through a quality-check pipeline combining rules and model scoring.Covers ​​8 languages​​ across ​​4 modalities​​, totaling ​​11.5 million entries​​, refined to industrial-grade quality for "ready-to-use" applications.
    Download Links​​: 5 languages (Arabic, Russian, Korean, Vietnamese, Thai)3 languages (Serbian, Hungarian, Czech).

🔥🔥🔥OpenDataLab Provide ecology for high-quality datasets for community. It provides:

● High-speed and simple way to access open datasets
● 7700+ Large scale and high-quality open datasets for large model
● 1200+ Open datasets for Computer Vision
● 200+ Open datasets by CVPR
● Categorized datasets for hot topics

● Data acquisition toolkits supporting large datasets
● Data acquisition toolkits supporting kinds of tasks
● Open source intelligent Toolbox for Labeling

● Format standardization
● DSDL: Dataset Description Language
● Define a CV dataset by DSDL
● OpenDataLab Standardized 100+ CV Datasets

Check our tutorials videos (in Chinese) to get started.


📣 We have upgraded and launched the function of authors uploading datasets independently. We hereby invite you to participate in using it to better promote your open source datasets, AI research results, etc., so that more people can access, obtain and use your dataset.

This is an introduction to the dataset autonomous upload function 【help doc】,You can create and share your dataset according to our guidelines. 💪

If you have any questions or obstacles, please feel free to contact us OpenDataLab@pjlab.org.cn.

Popular repositories Loading

  1. MinerU MinerU Public

    A high-quality tool for convert PDF to Markdown and JSON.一站式开源高质量数据提取工具,将PDF转换成Markdown和JSON格式。

    Python 38.3k 3.1k

  2. PDF-Extract-Kit PDF-Extract-Kit Public

    A Comprehensive Toolkit for High-Quality PDF Content Extraction

    Python 8.1k 590

  3. DocLayout-YOLO DocLayout-YOLO Public

    DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception

    Python 1.4k 113

  4. labelU labelU Public

    Data annotation toolbox supports image, audio and video data.

    Python 1.3k 132

  5. LabelLLM LabelLLM Public

    The Open-Source Data Annotation Platform

    TypeScript 861 90

  6. OmniDocBench OmniDocBench Public

    [CVPR 2025] A Comprehensive Benchmark for Document Parsing and Evaluation

    Python 588 50

Repositories

Showing 10 of 48 repositories
  • MinerU Public

    A high-quality tool for convert PDF to Markdown and JSON.一站式开源高质量数据提取工具,将PDF转换成Markdown和JSON格式。

    Python 38,273 AGPL-3.0 3,148 111 6 Updated Jul 5, 2025
  • labelU-Kit Public

    Data annotation component library --provided as NPM packages

    TypeScript 111 Apache-2.0 35 1 1 Updated Jul 5, 2025
  • labelU Public

    Data annotation toolbox supports image, audio and video data.

    Python 1,261 Apache-2.0 132 23 (2 issues need help) 1 Updated Jul 4, 2025
  • OmniDocBench Public

    [CVPR 2025] A Comprehensive Benchmark for Document Parsing and Evaluation

    Python 588 Apache-2.0 50 30 2 Updated Jul 4, 2025
  • 2 2 0 0 Updated Jul 2, 2025
  • opendatalab-datasets Public

    datasets resource

    117 11 3 0 Updated Jul 1, 2025
  • .github Public
    1 2 0 0 Updated Jul 1, 2025
  • OHR-Bench Public

    (ICCV 2025) OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

    Python 78 12 0 0 Updated Jul 1, 2025
  • skydiffusion Public

    [ICCV 2025] The official implementation of the paper “Street-to-Satellite Image Synthesis with Diffusion Models and BEV Paradigm”

    Python 55 Apache-2.0 2 4 0 Updated Jun 27, 2025
  • Vis3 Public

    OSS browser based on s3. 一个基于 S3 的数据(json / jsonl / html / md等)可视化工具。👇 Try online.

    TypeScript 35 Apache-2.0 4 0 0 Updated Jun 27, 2025

Most used topics

Loading…