Different results after version change

[augusta8]

Hi there -

We have been running the same code with small variations over the course of this year and last and have begun getting different results when we run one iteration of code across different computers, since some members of the team updated R and reinstalled MatchIt. The before and after matching baseline characteristics are the same across versions, but the patient numbers matched differ. When we extract a patient list (of matched patients), we get a different number to in the MatchIt output also.

Is there some adjustment being applied to the patient numbers? We wondered about a weighting. Any ideas about why this is happening would be much appreciated. Thank you!

Augusta

[ngreifer]

We did just update MatchIt and in doing so completely rewrote most of the underlying code. Some changes were made to enhance reproducibility moving forward, but may have caused changes from past versions. The changes you are seeing may depend on what type of matching you were performing. You can always re-install previous versions of MatchIt and continue on with those older versions for this project.

What method of matching are you using (and with which options)? Also, just to clarify, you are saying that when you run match.data() on the matchit() output on one single machine to create the matched dataset, the number of rows in that dataset is different from the sample size reported in the call to summary()? Or are you saying that the output of match.data() differs across computers based on the version of MatchIt used, not only in the composition but also in the size?

[augusta8]

Thank you for your quick reply. We will try reinstalling an the older version to see.

Please see the code below for the method we are using. We have been generating the list using the match matrix, and yes, on one computer the resulting number of individuals listed when we send that to an Excel file then differs from the number matched reported by summary().

m.out <- matchit(whole_data$STUDY ~ whole_data$X + whole_data$Y + whole_data$Z, data=whole_data, method="nearest", ratio=1, replace=TRUE)
summary(m.out)
m.dataw <- match.data(m.out)
m.dataw.matrix <- m.out$match.matrix
write.xlsx(m.dataw.matrix, file = "XXXX.xlsx")

[ngreifer]

I see. This is a misunderstanding about match.matrix. Each row of match.matrix corresponds to a treated unit, with the contents of the row being the control unit(s) matched to the corresponding treated unit. When you match with replacement (i.e., with replace = TRUE), multiple control units are matched to the same treated unit, so they will appear multiple times in match.matrix but only once in the match.data() output and will only be counted once in the summary() output.

As an extreme example, imagine there was a single control unit that was the nearest neighbor to every treated unit. In match.matrix, every row would have that control unit in it. There would be n1 rows in match.matrix (where n1 is the number of treated units), and a total of n1*2 filled cells in the spreadsheet, perhaps leading you to think there should be n1*2 rows in the match.data() output. But actually only n1+1 units have been matched, because the sole control unit was matched to every treated unit. summary() would display n1 treated units and 1 control unit, and that same composition would appear in match.data() output.

If you want a dataset that has one row for each time a unit appears in a match, you need to use get_matches() instead of match.data() (which only works in the newer version of MatchIt). Indeed, get_matches() was designed precisely for this scenario. In the extreme example I described, the get_matches() output would contain n1*2 rows, one for each time a unit appeared in a match. Each treated unit appears once, and the one matched control unit appears n1 times (because it is matched to all n1 treated units). How to analyze data after matching with replacement is described on the MatchIt website here. This dataset will contain patient pair membership and patient ID for each row, with multiple rows corresponding to the same control patients because they are matched to multiple treated units.

Regarding the discrepancies across versions, this may be due to the fact that old versions of MatchIt randomly selected units to match when there was a tie. In the new version, matchit() chooses in the order of the dataset so that the result is the same each time without setting a seed, thereby enhancing reproducibility. That's the only explanation I can see for the difference between old and new versions of MatchIt when using the matching method you used.

[augusta8]

Sure, I understand this - once we exported the list to Excel, we have been generating a de-duplicated list and determining how many times a control individual was matched. However, the deduplicated number still did not match the summary() output. We will try using get_matches() in future, that's a great option.

It sounds as though your last point may be the explanation for our problem though. We are currently trying to reinstall older versions of MatchIt on the two machines where we have been getting these odd results, to see if they match the non-updated machine's results.

[ngreifer]

Please also let me know if this is occurring in the new version of MatchIt. If it's only occurring in the old version, then we have a solution (upgrade). If it's happening in the new version I want to make sure this bug is fixed! Sorry for the confusion this has caused.

[augusta8]

We have so far not been able to solve the issue with the results being different on different machines. We do have the latest versions of MatchIt on two computers, but an older version on the other. It is not clear which version is the 'correct' one without knowing what is causing the disparity. I will let you know if we find a solution, thank you for your help.

[joatob]

Hello,
I have the same issue with the newest version and version 3.0.2. I'm in the middle of a longer experiment and I've been running the same code for several months testing different parameters, so the reproducibility is much needed. My observations are divided into two sets 100 times in the loop with the following code:
matched = matchit(as.formula(my_formula), data = ds, ratio = 1)
ds.matched = match.data(matched)
I set seed before the loop starts. I used to obtain a different set of 100 divisions (every time I ran the code the same thanks to the seed) using MatchIt 3.0.2 than now.
I installed the previous version again and the divisions are now the same as they used to be. Could you please explain why that happens? Is it just because now matchit() chooses in the order of the dataset, as you mentioned before?
Thank you,
Joanna

[ngreifer]

@joatob, when nearest neighbor matching without replacement and without a caliper, matchit() 3.0.2 would randomly break ties. Now it does not. That is the only difference in behavior in your scenario. A tie could occur when two treated units have the same propensity score, in which case one would randomly be chosen to be matched first, or when two control units are equally close to a given treated unit, in which case one would be randomly chosen to be matched. In order to ensure reproducibility, users had to set a seed. In version 4.0.0 and up, ties are no longer broken randomly but rather based on the order of the data. While this is an arbitrary choice, it guarantees reproducibility without a seed when using the same dataset.

This comes at the expense of old results no longer being reproducible with newer versions of MatchIt, but that is a choice all package developers have to make when fixing potentially problematic old behavior. I recommend you (and any users having the same issue) simply use the old version of MatchIt until your project is finished and then upgrade to the new version, or restart your project with the new version, which will enhance reproducibility moving forward.

[tomiko20]

Hello,
I too found that by running my R code on Jupyter Notebook, my results differed recently from what I had seen weeks ago. After some troubleshooting, I found that by installing version 3.0.1 first (code below),

## FIRST, install version 3.0.1
require(devtools)
install_version("MatchIt", version='3.0.1', repos = "http://cran.us.r-project.org")
library(MatchIt)
packageVersion("MatchIt")

and then installing a later version of Matchit, it appears my code works again to produce the same results as before.

[ngreifer]

I encountered this package, groundhog, that encourages reproducibility by making it easier to install the version of a package available at a certain date: http://groundhogr.com/

I recommend using this with your analyses to be able to use older versions of MatchIt.

For example, to reproduce an analysis using a version of MatchIt that was installed on 2020-01-17, you can run

library(groundhog)
groundhog.library("MatchIt", "2020-01-17")

If you have this at the beginning of your scripts, you don't need to worry about keeping track of versions.