acc_form.Rmd

---
output:
  pdf_document: default
  html_document: default
---
<!--HOW TO COMPLETE THIS FORM:-->

<!--
1. Checkboxes in this document appear as follows: 

- [ ] This is a checkbox 

To check a checkbox, replace [ ] by [x], as follows: 

- [x] This is a checked checkbox 

Note that older versions of RStudio (versions lower than 1.3) may not create a formatted checkbox but will leave the original characters, i.e., literally "[ ]" or "[x]". It's fine to submit a PDF in this form.
 
2. For text answers, simply type the relevant text in the areas indicated. A blank line starts a new paragraph. 
 
3. Comments (like these instructions) provide additional instructions throughout the form. There is no need to remove them; they will not appear in the compiled document. 

4. If you are comfortable with Markdown syntax, you may choose to include any Markdown-compliant formatting in the form. For example, you may wish to include R code chunks and compile this document in R Markdown.
-->

This form documents the artifacts associated with the article (i.e., the data and code supporting the computational findings) and describes how to reproduce the findings.


# Part 1: Data

- [ ] This paper does not involve analysis of external data (i.e., no data are used or the only data are generated by the authors via simulation in their code).

<!--
If box above is checked and if no simulated/synthetic data files are provided by the authors, please skip directly to the Code section. Otherwise, continue.
-->

- [x] I certify that the author(s) of the manuscript have legitimate access to and permission to use the data used in this manuscript.

<!-- If data are simulated using random number generation, please be sure to set the random number seed in the code you provide -->

## Abstract

<!--
Provide a short (< 100 words), high-level description of the data
-->

## Availability


- [ ] Data **are** publicly available.
- [x] Data **cannot be made** publicly available.

If the data are publicly available, see the *Publicly available data* section. Otherwise, see the *Non-publicly available data* section, below.

### Publicly available data

- [ ] Data are available online at:

- [ ] Data are available as part of the paper’s supplementary material.

- [ ] Data are publicly available by request, following the process described here:

- [ ] Data are or will be made available through some other mechanism, described here:


<!-- If data are available by request to the authors or some other data owner, please make sure to explain the process of requesting access to the data. -->

### Non-publicly available data

The data that support the findings of this study are available from a ride-hailing platform. Restrictions apply to the availability of these data, which were used under license for this study. 

## Description

### File format(s)

<!--
Check all that apply
-->
- [ ] CSV or other plain text.
- [ ] Software-specific binary format (.Rda, Python pickle, etc.): pkcle
- [ ] Standardized binary format (e.g., netCDF, HDF5, etc.): 
- [ ] Other (please specify):

### Data dictionary

<!--
A data dictionary provides information that allows users to understand the meaning, format, and use of the data.
-->

- [ ] Provided by authors in the following file(s):
- [ ] Data file(s) is(are) self-describing (e.g., netCDF files)
- [ ] Available at the following URL: 

### Additional Information (optional)

<!-- 
OPTIONAL: Provide any additional details that would be helpful in understanding the data. If relevant, please provide unique identifier/DOI/version information and/or license/terms of use.
-->

# Part 2: Code

## Abstract

The proposed estimators are implemented in `opeuc.py` 
including direct estimator, importance sampling estimator and confounded off-policy estimator. The other code are implemented for estimating nuisance parameters, sampling, and numerical experiments. 

## Description

### Code format(s)

<!--
Check all that apply
-->
- [x] Script files
    - [ ] R
    - [x] Python
    - [ ] Matlab
    - [ ] Other: 
- [x] Package
    - [ ] R
    - [x] Python
    - [ ] MATLAB toolbox
    - [ ] Other: 
- [ ] Reproducible report 
    - [ ] R Markdown
    - [ ] Jupyter notebook
    - [ ] Other:
- [ ] Shell script
- [ ] Other (please specify): 

### Supporting software requirements

#### Version of primary software used
                
Python version 3.7.8

#### Libraries and dependencies used by the code
        
- `numpy`==1.20.3
- `scikit-learn`==0.24.2
- `scipy`==1.6.3
- `tensorflow-cpu`==2.6.0
- `pandas`==1.2.4

### Supporting system/hardware requirements (optional)

Windows 11, Intel(R) Core(TM) i9-9940X CPU @ 3.30GHz   3.31 GHz, 48.0 GB

### Parallelization used

- [x] No parallel code used
- [ ] Multi-core parallelization on a single machine/node
    - Number of cores used: 
- [ ] Multi-machine/multi-node parallelization 
    - Number of nodes and cores used: 

### License

- [ ] MIT License (default)
- [ ] BSD 
- [x] GPL v3.0
- [ ] Creative Commons
- [ ] Other: (please specify)


### Additional information (optional)

<!--
OPTIONAL: By default, submitted code will be published on the JASA GitHub repository (http://github.com/JASA-ACS) as well as in the supplementary material. Authors are encouraged to also make their code available in a public code repository, such as on GitHub, GitLab, or BitBucket. If relevant, please provide unique identifier/DOI/version information (e.g., a Git commit ID, branch, release, or tag). If the code and workflow are provided together, this section may be omitted, with information provided in the "Location" section below.
-->

# Part 3: Reproducibility workflow

<!--
The materials provided should provide a straightforward way for reviewers and readers to reproduce analyses with as few steps as possible. 
-->

## Scope

The provided workflow reproduces:

- [ ] Any numbers provided in text in the paper
- [x] The computational method(s) presented in the paper (i.e., code is provided that implements the method(s))
- [ ] All tables and figures in the paper
- [x] Selected tables and figures in the paper, as explained and justified below:

- Figure 2
- Figure 3
- Figure S1
- Figure S2
- Figure S3
- Table S1
- Table S2

## Workflow

### Location

The workflow is available:

<!--
Check all that apply, and in the case of a Git repository include unique identifier, such as specific commit ID, branch, release, or tag.
-->
- [ ] As part of the paper’s supplementary material.
- [x] In this Git repository: the `main` branch
- [ ] Other (please specify):

<!--
Indicate where the materials (generally including the code, unless in a separate location and indicated in the previous section) are available. We strongly encourage authors to place their materials (but not large datasets) in a Git repository hosted on a site such as GitHub, GitLab, or BitBucket. If the repository is private during the review process, please indicate the location where it will be available publicly upon publication, and also include the materials as a zip file (e.g., obtained directly from the Git hosting site) as supplementary materials.
-->


### Format(s)

<!--
Check all that apply
-->
- [ ] Single master code file 
- [ ] Wrapper (shell) script(s)
- [ ] Self-contained R Markdown file, Jupyter notebook, or other literate programming approach
- [x] Text file (e.g., a readme-style file) that documents workflow
- [ ] Makefile
- [ ] Other (more detail in *Instructions* below)

### Instructions

- `sim_robust.py` --> Figure 2
- `sim_trajectory_compare_multdim.py` & `sim_time_compare_multdim.py` --> Figure 3
- `sim_trajectory_compare.py` & `sim_time_compare.py` --> Figure S3
- `sim_ratiolearner_compare.py` --> Table S1, Table S2, Figure S1
- `sim_ratio_features_number_compare.py` --> Figure S2

### Expected run-time

Approximate time needed to reproduce the analyses on a standard desktop machine:

- [ ] < 1 minute
- [ ] 1-10 minutes
- [ ] 10-60 minutes
- [x] 1-8 hours
- [ ] > 8 hours
- [ ] Not feasible to run on a desktop machine, as described here:

### Additional information (optional)

<!--
OPTIONAL: Additional documentation provided (e.g., R package vignettes, demos or other examples) that show how to use the provided code/software in other settings.
-->

# Notes (optional)

<!--
OPTIONAL: Any other relevant information not covered on this form. If reproducibility materials are not publicly available at the time of submission, please provide information here on how the reviewers can view the materials.
-->