**This Graphics is retrieved from Google Search**
This page contains all the Datasets and Code bases (experiments and evaluations) involved in experimenting and establishing our newly proposed LDR-HDR dataset (GTA-HDR) for Deep Learning models. In this work, we propose GTA-HDR, a large-scale synthetic dataset for HDR image reconstruction, sampled from the photo-realistic (i.e., HDR-10, link enabled game Grand Theft Auto V (GTA-V) by Rockstar Games.
The official repository of the paper with supplementary: GTA-HDR!!
This project is a collaboration between Monash University, Malaysia campus and Monash University, Melbourne, Australia.
Project Members -
Hrishav Bakul Barua (Monash University and TCS Research, Kolkata, India),
Kalin Stefanov (Monash University, Melbourne, Australia),
KokSheik Wong (Monash University, Malaysia),
Abhinav Dhall (Indian Institute of Technology (IIT) Ropar, India and Flinders University, Adelaide, Australia) , and
Ganesh Krishnasami (Monash University, Malaysia).
This work is supported by the prestigious Global Excellence and Mobility Scholarship (GEMS), Monash University. This research is also supported, in part, by the E-Science fund under the project: Innovative High Dynamic Range Imaging - From Information Hiding to Its Applications (Grant No. 01-02-10-SF0327).
Given that the performance of any data-driven learning- based method for HDR image reconstruction largely depends on the size and diversity of the datasets used for development, there is a significant gap in the publicly available datasets required to advance this research direction. Moreover, currently, there are no available datasets that adequately address the problem of no- reference HDR image quality assessment (other link1, link2), which demands vast collections of ground truth HDR (HDRGT) and distorted HDR (HDRDis) pairs. In summary, there is a substantial research gap pertaining to benchmark datasets needed to advance the research on HDR image reconstruction, motivating the creation of an appropriate large-scale dataset.
Tone mapping is the process of mapping the colors of HDR images capturing real-world scenes with a wide range of illumination levels to LDR images appropriate for standard displays with limited dynamic range. Inverse tone mapping is the reverse process accomplished with either traditional (non-learning) methods or data-driven learning-based approaches.
The proposed GTA-HDR dataset addresses the identified gaps in the current publicly available datasets for HDR image reconstruction, including:
a) GTA-HDR is a large-scale (i.e., 40K ground truth HDR images) synthetic dataset sampled from the GTA-V video game data which utilizes ray tracing technology that simulates the physics behind light and shadows.
b) GTA-HDR includes HDR images with sufficient resolution (i.e., 512 × 512 and 1024 × 1024).
c) GTA-HDR includes HDR images capturing a diverse set of scenes including, different locations (e.g., indoor, urban, rural, in-the-wild), different lighting conditions (e.g., morning, midday, evening, night), and different weather and season conditions (e.g., summer, winter, snowy, rainy, sunny, cloudy).
NOTE: The official GTA-HDR Benchmark Dataset is releasing soon!!
| Steps | Details/Link |
|---|---|
| 1(a). GTA-V has built-in HDR-10 support for displaying video sequences on HDR displays | The details can be found in Link1, Link2 |
| 1(b). We used Script Hook V plugin to capture HDR frames from the GTA-V game-play sequences | The details can be found in Link1, Link2 |
| 1(c). Our pipeline is inspired by the code base adopted from RenderDoc for Game data (IEEE TIP 2021) | The details can be found in Link1, Link2 |
| 2. We removed frames that are similar to the previous or next frames in the sequence to avoid unnecessary increase in dataset size | We find the Frame Similarity using Chamfer Similarity Metric: Link |
| 3(a). We performed transformations on the original LDR images to generate multiple exposure LDR images (i.e., exposure values EV 0, ±1, ±2, ±3, and ±4) and different contrast levels | For exposure: Link and contrast: Link |
| 3(b). We also generated distorted HDR images by randomly utilizing state-of-the-art methods | Some of the methods: M1, M2, M3 |
We stored the final HDR and LDR images in ".hdr" and ".png" formats, respectively
NOTE: Kindly fillup the agreement form to get access to the novel GTA-HDR Dataset used in this work.
Please fillup the Dataset Agreement Form and mail to Hrishav Bakul Barua (hbarua@acm.org) to get access.
GTA-HDR dataset scene diversity. Samples from the GTA-HDR dataset with multiple variations in location, weather, objects and time. The scene diversity ensures a thorough coverage of pixel colors, brightness, and luminance.
GTA-HDR dataset image diversity. For any image-to-image translation dataset, it is important to include a sufficient amount of samples with diverse range of color hues, saturation, exposure, and contrast levels. The final set of images in the dataset amounts to a total of 40K × 25 = 1M LDR, 40K HDR, and 40K distorted HDR images.
ACM TOG 2017 | HDRCNN - HDR image reconstruction from a single exposure using deep CNNs | Code
ACM TOG 2017 | DrTMO - Deep Reverse Tone Mapping | Code
GlobalSIP 2019 | FHDR - HDR Image Reconstruction from a Single LDR Image using Feedback Network | Code
CVPR 2020 | SingleHDR - Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline | Code
IEEE TIP 2021 | HDR-GAN - HDR Image Reconstruction from Multi-Exposed LDR Images with Large Motions | Code
WACV 2023 | SingleHDR - Single-Image HDR Reconstruction by Multi-Exposure Generation | Code
APSIPA 2023 | ArtHDR-Net - Perceptually Realistic and Accurate HDR Content Creation | Code
HistoHDR-Net - HistoHDR-Net: Histogram Equalization for Single LDR to HDR Image
Translation| Code
VPQM 2015 | HDR-Eye - Visual attention in LDR and HDR images | Link
ICCV 2017 | City Scene, site1 - Learning high dynamic range from outdoor panoramas | Link
City Scene, site2- The Laval HDR sky database | Link
ACM TOG 2017 | Kalantari et al. - Deep high dynamic range imaging of dynamic scenes | Link
ACM TOG 2017 | Endo et al. - Deep reverse tone mapping | Link
ACM TOG 2017 | Eilertsen et al. - HDR image reconstruction from a single exposure using deep CNNs | Link
IEEE Access 2018 | Lee et al. - Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image | Link
IEEE TIP 2018 | Cai et al. - Learning a deep single image contrast enhancer from multi-exposure images | Link
IEEE ICCP 2019 | Prabhakar et al. - A fast, scalable, and reliable deghosting method for extreme exposure fusion | Link
IEEE Access 2020 | LDR-HDR Pair - Dynamic range expansion using cumulative histogram learning for high dynamic range image generation | Link
CVPR 2020 | HDR-Synth & HDR-Real - Single-image HDR reconstruction by learning to reverse the camera pipeline | Link
SIGGRAPH 2022 | SI-HDR - Comparison of single image hdr reconstruction methods — the caveats of quality assessment | Link
Common link to some commonly used datasets - RAISE (ACM MMSys'15), HDR-Synth, HDR-Real, and HDR-Eye
Another huge dataset by Deep-SR-ITM, Kim et al. (ICCV 2019) (not used in our evaluation due to reasons stated in the paper)
Dataset: Sen et al. (ACM TOG 2012) (not used in our evaluation due to reasons stated in the paper)
Dataset: Trusun et al. (Wiley CGF 2016) (not used in our evaluation due to reasons stated in the paper)
IEEE TPAMI 2022 | Deep Learning for HDR Imaging: State-of-the-Art and Future Trends | Link
Elsevier DSP 2023 | High Dynamic Range Image Tone Mapping: Literature review and performance benchmark | Link
| Dataset | Type | #HDRGT | Resolution | In-the-wild | HDRDis | Scene diversity | Image diversity |
|---|---|---|---|---|---|---|---|
| HDR-Eye (2015) | Synthetic | 46 | 512✗512 | ❌ | ❌ | ❌ | ❌ |
| City Scene (2017) | Mixed | 41222 | 128✗64 | ❌ | ❌ | ✅ | ❌ |
| Kalantari et al. (2017) | Real | 89 | 1500✗1000 | ❌ | ❌ | ❌ | ❌ |
| Endo et al. (2017) | Synthetic | 1043 | 512✗512 | ❌ | ❌ | ❌ | ❌ |
| Eilertsen et al. (2017) | Synthetic | 96 | 1024✗768 | ❌ | ❌ | ❌ | ❌ |
| Lee et al. (2018) | Synthetic | 96 | 512✗512 | ❌ | ❌ | ❌ | ❌ |
| Cai et al. (2018) | Synthetic | 4413 | 3072✗1620 | ❌ | ❌ | ❌ | ❌ |
| Prabhakar et al. (2019) | Real | 582 | 1200✗900 | ❌ | ❌ | ❌ | ❌ |
| LDR-HDR Pair (2020) | Real | 176 | 1024✗1024 | ❌ | ❌ | ❌ | ❌ |
| HDR-Synth & HDR-Real (2020) | Mixed | 20537 | 512✗512 | ❌ | ❌ | ❌ | ✅ |
| SI-HDR (2022) | Real | 181 | 1920✗1280 | ❌ | ❌ | ✅ | ❌ |
| GTA-HDR (ours) (2024) | Synthetic | 40000 | 512✗512 | ✅ | ✅ | ✅ | ✅ |
Impact of the GTA-HDR dataset on the performance of the state-of-the-art in HDR image reconstruction. Impact of the GTA-HDR dataset on the performance of the state-of-the-art in HDR image reconstruction. R: Real data combines the datasets proposed in Kalantari et al. (2017), Prabhakar et al. (2019), LDR-HDR Pair (2020) and the real images from the datasets proposed in City Scene (2017); R ⊕ S: This combination contains the mixed datasets (including both real and synthetic data) proposed in City Scene (2017) and the real datasets proposed in Kalantari et al. (2017), Prabhakar et al. (2019), LDR-HDR Pair (2020); GTA-HDR: Proposed synthetic dataset; E2E: End-to-end training; FT: Finetuning of the original pre-trained models. The performance of all methods is evaluated on a separate dataset proposed in HDR-Synth & HDR-Real (2020).
For more details and experimental results please check out the paper!!
HDR images reconstructed with and without GTA-HDR as part of the training dataset, along with the RGB histograms and KL-divergence values. Base: HDR images reconstructed with ArtHDR-Net trained without GTA-HDR data; Ours: HDR images reconstructed with ArtHDR-Net trained with GTA-HDR data; GT: Ground truth.
For more details and experimental results please check out the paper!!
A page for awsome Deep Learning based HDR Reconstruction models: Link1, Link2
A page for awsome Deep Learning based Underexposed Image Enhancement models: Link
If you find our work (i.e. the code, the theory/concept, or the dataset) useful for your research or development activities, please consider citing our work as follows:
@article{barua2024gta,
title={GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction},
author={Barua, Hrishav Bakul and Stefanov, Kalin and Wong, KokSheik and Dhall, Abhinav and Krishnasamy, Ganesh},
journal={arXiv preprint arXiv:2403.17837},
year={2024}
}
----------------------------------------------------------------------------------------
Copyright 2024 | All the authors and contributors of this repository as mentioned above.
----------------------------------------------------------------------------------------
Please check the License Agreement.