Merge pull request #154 from camriddell/remark-cli

REMARK cli.py
econ-ark · Apr 3, 2024 · 717319b · 717319b
2 parents 089dae2 + 70839b0
commit 717319b
Show file tree

Hide file tree

Showing 30 changed files with 528 additions and 39 deletions.
diff --git a/README.md b/README.md
@@ -58,3 +58,92 @@ Jupyter notebook(s) should:
 ## Differences with DemARK
 
 The key difference with the contents of the [DemARK](https://github.com/econ-ark/DemARK) repo is that REMARKs are allowed to rely on the existence of local files and subdirectories (figures; data) at a predictable filepath relative to the location of the root.
+
+## For Maintainers
+
+**Command Line Interface** `cli.py`
+
+`cli.py` is an automated tool that facilitates:
+- cloning of REMARK repositories
+- linting (detection of missing files from a given REMARK)
+- building conda environments/docker images
+    - uses `conda`/`repo2docker` under the hood
+- executing `reproduce.sh` scripts within the built environments.
+
+All artifacts generated by `cli.py` are stored in a newly created `_REMARK` folder.
+
+1. Once you clone a REMARK you'll be able to find its contents inside of `_REMARK/repos/…`
+2. Once you build/execute a REMARK you'll be able to find a corresponding log
+file from that process inside of `_REMARK/logs/…`
+
+`cli.py` has built-in parallelization specified by the `-J` flag for many actions.
+
+### Requirements
+
+- python 3.9 or newer.
+- contents `requirements.txt`
+
+### Action
+
+**Clone/Pull**
+
+pulling REMARKs (these are populated in the  `_REMARKS` folder)
+
+```bash
+python cli.py pull --all         # git clone all REMARKS
+python cli.py pull {remark_name} # git clone one or more REMARK(s)
+```
+
+**Lint**
+
+Shows what files are missing from given REMARK(s). The linter uses the
+file-tree print out from STANDARD.md and compares it to the current files found
+in the currently cloned REMARK(s).
+
+```bash
+python cli.py lint --all # detect missing files from all REMARKs
+python cli.py lint {remark_name} # detect missing files from one or more REMARK(s)
+```
+
+**Build**
+
+Building conda environments and/or docker images.
+
+```bash
+python cli.py build conda --all          # build conda environments for all REMARKs (stored as a `condaenv` folder inside the cloned REMARK repo)
+python cli.py build docker --all         # build docker images for all REMARKs (stored as a `condaenv` folder inside the cloned REMARK repo)
+python cli.py build conda {remark_name}  # build conda environments for one or more REMARK(s)
+python cli.py build docker {remark_name} # build docker image(s) for one or more REMARK(s)
+```
+
+The primary difference between `conda` and `docker` for builds are that `docker` will be more flexible for multilanguage REMARKs. It leverages 
+repo2docker (same tool that mybinder uses) to create docker images from repositories.
+
+**Execute**
+
+Automated execution within built conda environments/docker containers.
+
+```bash
+python cli.py execute conda --all          # execute reproduce.sh via conda for all REMARKs
+python cli.py execute docker --all         # execute reproduce.sh via docker for all REMARKs
+python cli.py execute conda {remark_name}  # execute reproduce.sh via conda for one or more REMARK(s)
+python cli.py execute docker {remark_name} # execute reproduce.sh via docker for one or more REMARK(s)
+```
+
+*Both the build and execute subcommands have an optional --jobs argument to
+specify the number of jobs to run in parallel when building/executing.*
+
+**Logs/Summarize**
+
+```bash
+python cli.py logs # view most recent logs for all previous building/executing commands
+```
+
+**Clean/Remove**
+
+```bash
+python cli.py clean conda --all          # remove all built conda environments
+python cli.py clean docker --all         # remove all build docker images
+python cli.py clean conda {remark_name}  # remove conda environment(s) from specified REMARK(s)
+python cli.py clean docker {remark_name} # remove docker images built from specified REMARK(s)
+```
diff --git a/REMARKs/Aiyagari-Idiosyncratic.yml b/REMARKs/Aiyagari-Idiosyncratic.yml
@@ -0,0 +1,3 @@
+name: Aiyagari-Idiosyncratic
+remote: https://github.com/econ-ark/Aiyagari-Idiosyncratic
+title: Uninsured Idiosyncratic Risk and Aggregate Saving
diff --git a/REMARKs/BayerLuetticke.yml b/REMARKs/BayerLuetticke.yml
@@ -0,0 +1,4 @@
+name: BayerLuetticke
+remote: https://github.com/econ-ark/BayerLuetticke
+title: Solving heterogeneous agent models in discrete time with many idiosyncratic
+  states by perturbation methods
diff --git a/REMARKs/BlanchardPA2019.bib b/REMARKs/BlanchardPA2019.bib
diff --git a/REMARKs/BlanchardPA2019.yml b/REMARKs/BlanchardPA2019.yml
@@ -0,0 +1,3 @@
+name: BlanchardPA2019
+remote: https://github.com/econ-ark/BlanchardPA2019
+title: Public Debt and Low Interest Rates
diff --git a/REMARKs/BufferStock-LifeCycle.yml b/REMARKs/BufferStock-LifeCycle.yml
@@ -0,0 +1,3 @@
+name: BufferStock-LifeCycle
+remote: https://github.com/econ-ark/BufferStock-LifeCycle
+title: Buffer-Stock Saving and the Life Cycle/Permanent Income Hypothesis
diff --git a/REMARKs/BufferStockTheory.bib b/REMARKs/BufferStockTheory.bib
diff --git a/REMARKs/BufferStockTheory.yml b/REMARKs/BufferStockTheory.yml
@@ -0,0 +1,3 @@
+name: BufferStockTheory
+remote: https://github.com/econ-ark/BufferStockTheory
+title: BufferStockTheory
diff --git a/REMARKs/CGMPortfolio.yml b/REMARKs/CGMPortfolio.yml
@@ -0,0 +1,3 @@
+name: CGMPortfolio
+remote: https://github.com/econ-ark/CGMPortfolio
+title: 'REMARK: Consumption and Portfolio Choice Over the Life Cycle'
diff --git a/REMARKs/DistributionofWealthMPC.yml b/REMARKs/DistributionofWealthMPC.yml
@@ -0,0 +1,3 @@
+name: DistributionofWealthMPC
+remote: https://github.com/econ-ark/DistributionOfWealthMPC
+title: The distribution of wealth and the marginal propensity to consume
diff --git a/REMARKs/DurableConsumerType.yml b/REMARKs/DurableConsumerType.yml
@@ -0,0 +1,3 @@
+name: DurableConsumerType
+remote: https://github.com/econ-ark/DurableConsumerType_REMARK
+title: A Guide on Solving Non-convex Consumption-Saving Models
diff --git a/REMARKs/EndogeneousRetirement.yml b/REMARKs/EndogeneousRetirement.yml
@@ -0,0 +1,3 @@
+name: EndogeneousRetirement
+remote: https://github.com/econ-ark/EndogenousRetirement
+title: 'Endogenous Retirement: A Canonical Discrete-Continuous Problem'
diff --git a/REMARKs/EpiExp.yml b/REMARKs/EpiExp.yml
@@ -0,0 +1,3 @@
+name: EpiExp
+remote: https://github.com/econ-ark/EpiExp
+title: Epidemiological Expectations in Economics
diff --git a/REMARKs/GanongNoelUI.yml b/REMARKs/GanongNoelUI.yml
@@ -0,0 +1,3 @@
+name: GanongNoelUI
+remote: https://github.com/econ-ark/GanongNoelUI
+title: 'Consumer Spending during Unemployment: Positive and Normative Implications'
diff --git a/REMARKs/KrusellSmith.bib b/REMARKs/KrusellSmith.bib
diff --git a/REMARKs/KrusellSmith.yml b/REMARKs/KrusellSmith.yml
@@ -0,0 +1,3 @@
+name: KrusellSmith
+remote: https://github.com/econ-ark/KrusellSmith
+title: Income and Wealth Heterogeneity in the Macroeconomy
diff --git a/REMARKs/LiqConstr.yml b/REMARKs/LiqConstr.yml
@@ -0,0 +1,3 @@
+name: LiqConstr
+remote: https://github.com/econ-ark/LiqConstr
+title: Liquidity Constraints and Precautionary Saving
diff --git a/REMARKs/Pandemic.md b/REMARKs/Pandemic.md
@@ -68,6 +68,8 @@ keywords: # optional
 
 # Pandemic-Consumption-Response
 
+[![badge](https://img.shields.io/badge/Launch-Dashboard-579ACA.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFkAAABZCAMAAABi1XidAAAB8lBMVEX///9XmsrmZYH1olJXmsr1olJXmsrmZYH1olJXmsr1olJXmsrmZYH1olL1olJXmsr1olJXmsrmZYH1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olJXmsrmZYH1olL1olL0nFf1olJXmsrmZYH1olJXmsq8dZb1olJXmsrmZYH1olJXmspXmspXmsr1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olLeaIVXmsrmZYH1olL1olL1olJXmsrmZYH1olLna31Xmsr1olJXmsr1olJXmsrmZYH1olLqoVr1olJXmsr1olJXmsrmZYH1olL1olKkfaPobXvviGabgadXmsqThKuofKHmZ4Dobnr1olJXmsr1olJXmspXmsr1olJXmsrfZ4TuhWn1olL1olJXmsqBi7X1olJXmspZmslbmMhbmsdemsVfl8ZgmsNim8Jpk8F0m7R4m7F5nLB6jbh7jbiDirOEibOGnKaMhq+PnaCVg6qWg6qegKaff6WhnpKofKGtnomxeZy3noG6dZi+n3vCcpPDcpPGn3bLb4/Mb47UbIrVa4rYoGjdaIbeaIXhoWHmZYHobXvpcHjqdHXreHLroVrsfG/uhGnuh2bwj2Hxk17yl1vzmljzm1j0nlX1olL3AJXWAAAAbXRSTlMAEBAQHx8gICAuLjAwMDw9PUBAQEpQUFBXV1hgYGBkcHBwcXl8gICAgoiIkJCQlJicnJ2goKCmqK+wsLC4usDAwMjP0NDQ1NbW3Nzg4ODi5+3v8PDw8/T09PX29vb39/f5+fr7+/z8/Pz9/v7+zczCxgAABC5JREFUeAHN1ul3k0UUBvCb1CTVpmpaitAGSLSpSuKCLWpbTKNJFGlcSMAFF63iUmRccNG6gLbuxkXU66JAUef/9LSpmXnyLr3T5AO/rzl5zj137p136BISy44fKJXuGN/d19PUfYeO67Znqtf2KH33Id1psXoFdW30sPZ1sMvs2D060AHqws4FHeJojLZqnw53cmfvg+XR8mC0OEjuxrXEkX5ydeVJLVIlV0e10PXk5k7dYeHu7Cj1j+49uKg7uLU61tGLw1lq27ugQYlclHC4bgv7VQ+TAyj5Zc/UjsPvs1sd5cWryWObtvWT2EPa4rtnWW3JkpjggEpbOsPr7F7EyNewtpBIslA7p43HCsnwooXTEc3UmPmCNn5lrqTJxy6nRmcavGZVt/3Da2pD5NHvsOHJCrdc1G2r3DITpU7yic7w/7Rxnjc0kt5GC4djiv2Sz3Fb2iEZg41/ddsFDoyuYrIkmFehz0HR2thPgQqMyQYb2OtB0WxsZ3BeG3+wpRb1vzl2UYBog8FfGhttFKjtAclnZYrRo9ryG9uG/FZQU4AEg8ZE9LjGMzTmqKXPLnlWVnIlQQTvxJf8ip7VgjZjyVPrjw1te5otM7RmP7xm+sK2Gv9I8Gi++BRbEkR9EBw8zRUcKxwp73xkaLiqQb+kGduJTNHG72zcW9LoJgqQxpP3/Tj//c3yB0tqzaml05/+orHLksVO+95kX7/7qgJvnjlrfr2Ggsyx0eoy9uPzN5SPd86aXggOsEKW2Prz7du3VID3/tzs/sSRs2w7ovVHKtjrX2pd7ZMlTxAYfBAL9jiDwfLkq55Tm7ifhMlTGPyCAs7RFRhn47JnlcB9RM5T97ASuZXIcVNuUDIndpDbdsfrqsOppeXl5Y+XVKdjFCTh+zGaVuj0d9zy05PPK3QzBamxdwtTCrzyg/2Rvf2EstUjordGwa/kx9mSJLr8mLLtCW8HHGJc2R5hS219IiF6PnTusOqcMl57gm0Z8kanKMAQg0qSyuZfn7zItsbGyO9QlnxY0eCuD1XL2ys/MsrQhltE7Ug0uFOzufJFE2PxBo/YAx8XPPdDwWN0MrDRYIZF0mSMKCNHgaIVFoBbNoLJ7tEQDKxGF0kcLQimojCZopv0OkNOyWCCg9XMVAi7ARJzQdM2QUh0gmBozjc3Skg6dSBRqDGYSUOu66Zg+I2fNZs/M3/f/Grl/XnyF1Gw3VKCez0PN5IUfFLqvgUN4C0qNqYs5YhPL+aVZYDE4IpUk57oSFnJm4FyCqqOE0jhY2SMyLFoo56zyo6becOS5UVDdj7Vih0zp+tcMhwRpBeLyqtIjlJKAIZSbI8SGSF3k0pA3mR5tHuwPFoa7N7reoq2bqCsAk1HqCu5uvI1n6JuRXI+S1Mco54YmYTwcn6Aeic+kssXi8XpXC4V3t7/ADuTNKaQJdScAAAAAElFTkSuQmCC)](https://xhrtcvh6l53u.curvenote.dev/services/binder/v2/gh/econ-ark/Pandemic/HEAD?urlpath=/voila/render/Code/Python/dashboard.ipynb)
+
 This repository is a complete software archive for the paper "Modeling the Consumption Response to the CARES Act" by Carroll, Crawley, Slacalek, and White (2020). This README file provides instructions for running our code on your own computer, as well as adjusting the parameters of the model to produce alternate versions of the figures in the paper.
 
 ## References

diff --git a/REMARKs/Pandemic.yml b/REMARKs/Pandemic.yml
@@ -0,0 +1,3 @@
+name: Pandemic
+remote: https://github.com/econ-ark/Pandemic
+title: Modeling the Consumption Response to the CARES Act
diff --git a/REMARKs/PortfolioChoiceBlogPost.yml b/REMARKs/PortfolioChoiceBlogPost.yml
@@ -0,0 +1,3 @@
+name: PortfolioChoiceBlogPost
+remote: https://github.com/econ-ark/PortfolioChoiceBlogPost
+title: Optimal Financial Investment over the Life Cycle - Blog Post
diff --git a/REMARKs/RiskyContrib.yml b/REMARKs/RiskyContrib.yml
@@ -0,0 +1,3 @@
+name: RiskyContrib
+remote: https://github.com/econ-ark/RiskyContrib
+title: A Two-Asset Savings Model with an Income-Contribution Scheme
diff --git a/REMARKs/Sequential Endogenous Grid Method.yml b/REMARKs/Sequential Endogenous Grid Method.yml
@@ -0,0 +1,3 @@
+name: Sequential Endogenous Grid Method
+remote: https://github.com/alanlujan91/SequentialEGM
+title: 'EGM^n: The Sequential Endogenous Grid Method'
diff --git a/REMARKs/SolvingMicroDSOPs.md b/REMARKs/SolvingMicroDSOPs.md
@@ -49,4 +49,4 @@ keywords: # optional
 
 # Solution Methods for Microeconomic Dynamic Stochastic Optimization Problems
 
-These notes describe tools for solving microeconomic dynamic stochastic optimization problems, and show how to use those tools for efficiently estimating a standard life cycle consumption/saving model using microeconomic data. No attempt is made at a systematic overview of the many possible technical choices; instead, I present a specific set of methods that have proven useful in my own work (and explain why other popular methods, such as value function iteration, are a bad idea). Paired with these notes is Mathematica, Matlab, and Python software that solves the problems described in the text.
+These notes describe tools for solving microeconomic dynamic stochastic optimization problems, and show how to use those tools for efficiently estimating a standard life cycle consumption/saving model using microeconomic data.  No attempt is made at a systematic overview of the many possible technical choices; instead, I present a specific set of methods that have proven useful in my own work (and explain why other popular methods, such as value function iteration, are a bad idea).  Paired with these notes is Mathematica, Matlab, and Python software that solves the problems described in the text.
diff --git a/REMARKs/SolvingMicroDSOPs.yml b/REMARKs/SolvingMicroDSOPs.yml
@@ -0,0 +1,3 @@
+name: SolvingMicroDSOPs
+remote: https://github.com/econ-ark/SolvingMicroDSOPs
+title: Solution Methods for Microeconomic Dynamic Stochastic Optimization Problems
diff --git a/REMARKs/beyond-the-streetlight.yml b/REMARKs/beyond-the-streetlight.yml
@@ -0,0 +1,3 @@
+name: beyond-the-streetlight
+remote: https://github.com/dedwar65/beyond-the-streetlight
+title: Beyond the streetlight
diff --git a/REMARKs/cAndCwithStickyE.yml b/REMARKs/cAndCwithStickyE.yml
@@ -0,0 +1,3 @@
+name: cAndCwithStickyE
+remote: https://github.com/econ-ark/cAndCwithStickyE
+title: Sticky Expectations and Consumption Dynamics
diff --git a/REMARKs/ctDiscrete.yml b/REMARKs/ctDiscrete.yml
@@ -0,0 +1,3 @@
+name: ctDiscrete
+remote: https://github.com/econ-ark/ctDiscrete
+title: A Tractable Model of Buffer Stock Saving
diff --git a/STANDARD.md b/STANDARD.md
@@ -10,21 +10,26 @@ The PR should add a link to the repository to the Catalog (currently, in the REA
 
 ## The REMARK Standard
 
+```
+.
+|-- reproduce.sh
+|-- CITATION.cff
+`-- binder
+    `-- environment.yml
+```
+
 The REMARK's repository must:
  1. Have a [tagged release](https://docs.github.com/en/github/administering-a-repository/managing-releases-in-a-repository), the last commit before including it as a REMARK should be tagged with a 1.0 release.
  2. In that repository at that release, there must be:
-   - In either the top-level directory or a `binder/` directory, either:
-     - installation files for `pip`:
-       - a `runtime.txt` containing the name of a python version, e.g. `python-3.9.0`
-       - a `requirements.txt` file with pinned dependencies (such as created by the command `pip freeze > requirements.txt`), or...
-     - installation files for conda:
+   - There must be a `binder/` directory containing an `environment.yml`:
       - an `environment.yml` file with pinned dependencies
    - A `reproduce.sh` script that
      - Installs the requirements
      - Runs and reproduces all the results
-3. Include a valid CITATION.cff document with bibliographic
-metadata for the repository.
-
+3. Include a valid CITATION.cff document with bibliographic metadata for the repository.
+4. Subset of results that you are claiming to be reproducible.
+   - This could be specified as a directory?
+
 It is **strongly recommended** to include:
   - If reproduce.sh takes longer than a few minutes, a `reproduce_min.sh` that generates some interesting subset of results within a few minutes
   - A Jupyter notebook that exposits the material being reproduced.