speeding up poi2hdfe

[olidess]

Hi Segio,

I am using poi2hdfe, but it is very very slow. Is there anyway to speed the command?

Thanks,

Best,

Olivier

[sergiocorreia]

Have you tried out Tom Zylkin's ppml_panel_sg (check his code section)?

Stata command ... for "fast" estimation of Poisson models with many fixed effects. It is a bit similar to poi2hdfe by Paulo Guimaraes, only faster and more geared towards trade/gravity applications...

[victorialacaze]

Dear Sergio,
I´m estimating a gravity model of trade by comparing OLS (using reghdfe) and PPML (using T. Zylkin´s ppml_panel_sg) estimators. In both cases I have controlled for MRT and added FE.
When trying to run PPML, the following Stata error arises:
reghdfe error: data transformed with cache(save) requires cache(use) r(198)
Could you help me with this problem?
I apologize if my question is misplaced in this Issue.
Regards,
Victoria

[sergiocorreia]

A few questions:

1. what version of reghdfe are you using? (reghdfe, version)
2. Can you type set trace on, run the command, and paste or email me the output?

The error you got literally means what it says. You, or some program, is calling reghdfe with the cache(use) option, but reghdfe detected no previous use of cache(save). Without knowing what program called reghdfe and how, I wouldn't know.

Best,
S

[victorialacaze]

1. I´m using the 3.2.9 version, 21feb2016.
   Dependencies installed?

• ivreg2 not
• avar not
• tuples not

2. I´m attaching the output (the first part of it and the last part, since this was a long output and I didn´t manage to copy the whole text).
   Many thanks for your help!
   ppml output.docx

[sergiocorreia]

I'm searching and can't find the word reghdfe on the log file output. It also seems that the program worked without error.

BTW, if you want to save everything you can do something like

capture log close _all
log using "example"
capture noi ppml_sg ....
log close _all

[victorialacaze]

OK Sergio. I didn't know how to save the entire output. Perhaps the 'reghdfe' is in the missing (middle) part of my output. I will run the ppml command again and let you know. Another question for you: when I installed ppml_panel_sg I read that the command needs reghdfe, is it correct? Thanks for all Victoria El feb. 17, 2017 12:46, "Sergio Correia" <notifications@github.com> escribió:

…

I'm searching and can't find the word reghdfe on the log file output. It also seems that the program worked without error. BTW, if you want to save everything you can do something like capture log close _all log using "example" capture noi ppml_sg .... log close _all — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYo9FA9g0LlplcPt-cT1XfrWDcAh-Kuxks5rdcDCgaJpZM4J1M0T> .

[sergiocorreia]

AFAIK, yes, ppml_panel_sg uses reghdfe, but I don't recall exactly the details of how it uses it.

[victorialacaze]

Sergio, I sent you the output by email -it seems to be too large to be uploaded here.
Thanks,
Victoria

[sergiocorreia]

I've been scrolling through the (huge) log but can't find what the problem is.

BTW, have you spoken with the author of ppml_panel_sg? He might have an idea of what's going on..

[victorialacaze]

Yes I have and still waiting for his answer. I let you know when I have news of him Thank you for your time Victoria El feb. 21, 2017 12:14, "Sergio Correia" <notifications@github.com> escribió: I've been scrolling through the (huge) log but can't find what the problem is. BTW, have you spoken with the author of ppml_panel_sg? He might have an idea of what's going on.. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYo9FPi9HEQyPOSVwC0e4_BAYScCF6RGks5rev9tgaJpZM4J1M0T> .

[GiuliaSabbadini]

Dear Sergio,
I am Giulia and this is my first post. I am working on a paper studying the effects of Non-Tariff Measures (SPS and TBT) on Processed Food Exports. I have 149 countries, 13 years and 56 product lines for a total of 16,053,856 observations (exporter * importer * year *product). I constructed a dummy variable for both SPSjikt and TBTjikt notifications. The dummies pick up value 1 from year t and onwards if the importing country i imposed at least one sps/tbt measure on product line k in year t. The vector of the other variables includes standard gravity covariates.
I tried to use the ppml_panel_sg command by typing:
ppml_panel_sg trade sps tbt, ex(iso_o) im(iso_d) y(year) ind(HS) sym cluster(id)
where id=group(iso_o iso_d HS), the variable I used to set the panel dimension of the dataset.
However, this caused my variables of interest to be absorbed for perfect collinearity with the fixed effects. Therefore, I turned to -poi2hdfe-, where the fixed effects are importer-product and exporter-product FE. However, I'd like to ask of there is a way to speed up the process.
Thank you in advance for your time,
Giulia Sabbadini

[sergiocorreia]

Hi Giulia,

Yes, there is, but it's WiP. In case it's not that urgent, can you wait a week and then send me a reminder?

Best,
S

[GiuliaSabbadini]

Yes, sure. I'll send you a reminder in a week. Thanks again
Best,
Giulia

[GiuliaSabbadini]

Dear Sergio,
I am writing you as a reminder for the issued I raised last Monday regarding the -poi2hdfe- command. Meantime, I tried to estimate my regressions using the -xtpoisson- command with three sets of fixed effects:

xi: xtpoisson trade sps tbt controls i.importer-product i.exporter-product, robust i(importer-exporter-product) fe

I started running this piece of code by sectors, in order to deal with the high number of observations in my dataset. However, it is anyway extremely slow as well.

Thank you again for your time.
Best,
Giulia

[gknox79]

Dear Sergio,

Also interested in an answer to Giulia's question: is there a way to speed up the poi2hdfe command? Thanks for your help and contributions!

George

[arespitia325]

Dear Sergio,

Following this thread: is there a way to speed up the poi2hdfe command? Thanks for your help!

Alvaro

[sergiocorreia]

@gknox79 @arespitia325 sorry for the lack of updates but the short answer is that yes, there is.

However, we haven't released (it's still in "closed beta"), but feel free to shoot me an email.

-Sergio

[mhm1g11]

Dear Sergio,

currently using the poi2hdfe command intensively. Are there any soon to be expected news on ways to speed it up? Also, is there a way to get the startval speed up approach going, as explained in the help file? As it seems, the poi2hdfe does not allow for it.

Thanks a lot! KR

Mats

[sergiocorreia]

Yes ,we have a working demo (which is just getting updated). Shoot me an email and I'll send you the updated code.

[felixholub]

Besides the speed-up, will the update be able to include an exposure variable?

[sergiocorreia]

Just a quick update: we have a new program that does this (ppmlhdfe). It's in closed beta, so if you want to test it, shoot me an email (or Tom Zylkin, or Paulo Guimaraes).

[victorialacaze]

Great! I want to test it!!! El mar., 7 de ago. de 2018 16:55, Sergio Correia <notifications@github.com> escribió:

…

Just a quick update: we have a new program that does this (ppmlhdfe). It's in closed beta, so if you want to test it, shoot me an email (or Tom Zylkin, or Paulo Guimaraes). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYo9FE0jaXRvrSbSu9T5DWSy-fYDCtZoks5uOfC8gaJpZM4J1M0T> .