[Re] Assortative matching and search

[fgeoffroy]

Dear Editors,

I would like to request a review for the following replication.

Original articles:

• Shimer, R., and Smith, L. (2000). Assortative matching and search. Econometrica, 68(2), 343–369. http://doi.org/10.1111/1468-0262.00112
• Smith, L. (2006). The Marriage Model with Search Frictions. J. Political Econ., 114(6), 1124–1144. https://doi.org/10.1086/510440

PDF URL: https://github.com/fgeoffroy/-Re-Assortative-matching-and-search/blob/main/article/article.pdf
Metadata URL: https://github.com/fgeoffroy/-Re-Assortative-matching-and-search/blob/main/article/metadata.yaml
Code URL: https://github.com/fgeoffroy/-Re-Assortative-matching-and-search

Scientific domain: Game theory
Programming language: Python
Suggested editor:

[rougier]

Thank you for your submission and sorry for the delay. We'll assign an editor soon.

[rougier]

Actually, I can edit it.

[rougier]

@eddelbuettel @AurelienNioche Could you review this short submission in econometrics?

[eddelbuettel]

I could take a look but the self-selected field is game theory which is less my beef.

[AurelienNioche]

I agree that the label 'game theory' would be a better match than CS. I am also not familiar with Becker's model, but I can check that the code is doing what is supposed to do, and that I can reproduce the results.

[rougier]

Sorry for the mislabeling (I need to add more). For the revie,w you don't need to be specialist of the domain, only to appreciate if the results have been replicated as intended. If you can test the code, that's even better.

No firm deadline for review, I will remind you in one month.

[eddelbuettel]

Reproduction of Assortative matching and search for Shimer and Smith (2000) and Smith (2006)

The submission reproduces results from two seminal paper in game theory and matching. We find the reproduction to be faithful, replicable, and carefully executed. Minor comments follow.

Paper

The submitted paper is clearly written and introduces the scientific content, problem setting, and relevant references in five pages. It discusses the results carefullu, in particular the ability to reproduce all but one figure. The lack of reproducibility of the final is reasoned about in some detail revealing contact with the original author, lack of source code for that part of the original study as well some conjecture about possibly parameter misprints and possible approximation. Four pages present the reproduced figures which are also presented in the corresponding reproduction code repository.

Code

Usage and requirement of the submitted code are detailed appropriately. We appreciate that the required software packages are both small in numbers (only a Python interpreter plus two packages) and very mainstream (NumPy, Matplotlib). Yet we find the description overly stringent; Python code can be executed in environments other than Conda. The author lists Ubuntu 18.04 as the platform for his work; we suspect the code would have worked with the default setup. We reproduced everything under the (current) Ubuntu 20.10 release with versions 3.8.6 of Python, 1.18.4 of NumPy and 3.3.0 of Matplotlib. This would suggest that a less restrictive description might be suitable. If the author were truly concerned about versions, maybe a small Docker container with these versions could be provided too.

The code itself is commented and clearly enough written. My default editor and it elpy follow some coding guidelines and bemoan a lack of some def statements in main.py as well as an inappropriate number of lines---but I tend to find those nags to be overly stringent so this is not a real concern.

Figures

The main outcome of the reproduction are 20 pdf files. All appear to be identical to the submitted versions. As "eye-balling" figure has its limits, we made use of two R packages to i) convert the pdf files to png graphics and to ii) run a "visual test" comparing original and reproduction. Doing so is difficult, and has limits but we found the package used (visualTest, on GitHub but sadly not published to CRAN) appropriate (and we also use it in teaching and marking). The comparison script, in essence a loop over all all files with temporary file creation, conversion and comparison for each file, is below.

Disclaimer

This is my first review for ReScience, so please do lot me know if anything is insufficient or unclear. Game theory is also not something I have really touched since the graduate school days (quite some time ago) but as there are few other economists among the ReScience reviewer, I hope that this will still be of help.

Appendix

The ad-hoc R script follows. The two top-level directory assignment may need local adjustments.

## Compare two sets of pdf files for https://github.com/ReScience/submissions/issues/47
## Dirk Eddelbuettel, Feb 2021, released under GPL 2 or later

library(pdftools)		# on CRAN: to convert pdf to png
library(visualTest)		# at github.com/MangoTheCat/visualTest: to assess figure similarity

dirA <- "code/figures"          # soft link into author repo
dirB <- "re-run-figures"        # output of reproduction

pdffiles <- list.files(dirA, pattern="*.pdf")

for (f in pdffiles) {
    ## construct filenames in original and reproduced directories
    fileA <- file.path(dirA, f)
    fileB <- file.path(dirB, f)
    stopifnot(`no reproduced file found` = file.exists(fileB))

    ## create two tempfiles and produces png variants
    tfA <- tempfile(fileext=".png")
    tfB <- tempfile(fileext=".png")
    pdf_convert(fileA, format="png", filenames=tfA, verbose=FALSE)
    pdf_convert(fileB, format="png", filenames=tfB, verbose=FALSE)

    ## compare these png variants for exactness (approximately but still...)
    stopifnot(`disparity found` = isSimilar(tfA, tfB, exact=TRUE))

    ## report progress
    cat("Processed and compared", f, "without issues.\n")
}

cat("Done. No issues.\n")

[rougier]

@eddelbuettel Many thanks for the (fast) review that is exactly what is expected.

@fgeoffroy Could you address @eddelbuettel comments. I think the inclusion of the R script that @eddelbuettel indly offered would be a nice addition to your work. As for the less restrictive environment, I think it makes sense to loosen it a bit since your work is generic enough.

[fgeoffroy]

Dear all,

Thank you @eddelbuettel for your review and @rougier for your suggestions. I have updated the code, article and README file accordingly in the repository.

To answer @eddelbuettel's comments:

• I agree that the versioning description was too stringent, given that the packages used are very common. I have modified the README file so that the versioning aspect is no longer presented as a major issue. I have kept the Conda environment only for tractability reasons.

• In the main.py file, I assume that the lack of def statements that you mentioned corresponds to the use of lambda functions. Accordingly, I have replaced them for def statements in all python files.

• I have not changed the length of the main.py file, because I also do not think it is a real concern and, here, it is mostly designed to give information about each figure.

• I have also made minor aesthetic changes so that the python files follow the PEP-8 guidelines.

• I find the visual comparison idea very relevant, so I have included the R script provided by @eddelbuettel, as suggested by @rougier. Do you agree with this decision @eddelbuettel? It is mentioned that you are the author of this script in the README file, the article and in the script itself. To facilitate the comparison process, I have changed the folder structure: the article figures are now located in article/figures/ and the output of the main python script goes to code/figures/. The R script is named compare_images.R and is located in the code/ subfolder.

Lastly, thanks to @AurelienNioche's reply, I have realized that I had not mentioned the link between the replicated work and Becker's famous model. Thus, I have added a sentence and three relevant references in the Introduction.

[eddelbuettel]

That sounds good to me on all counts.

[rougier]

@AurelienNioche No pressure but if you send your review by the end of next week, I think we can hit a record as the fastest time for review :)