Questions about OG-USA release 0.5.9

[martinholmer]

The comment on OG-USA release 0.5.9 is this:

Works w/ TC 0.9.0, but breaks depending on Python environment.

How does OG-USA "break" and under what "Python environment"?

If it works with taxcalc package 0.9.0, then why not pin OG-USA to that version until you know OG-USA works with a more recent version of the taxcalc package?

You could do this by changing in both places the Python/conda.recipe/meta.yaml file to include:

    - taxcalc ==0.9.0

instead of the current

    - taxcalc

@jdebacker @rickecon @MattHJensen @PeterDSteinberg @jbcrail

[PeterDSteinberg]

@martinholmer I don't know what you mean by "breaks" - can't install it at all or has import problems after installation or fails later on in the middle of a calculation? It's hard to attack these problems without tracebacks and details about how it is being installed. For example, is the dev label being used, is it a new conda environment we are sure is free of prior installation-related complications, etc.?

OG-USA 0.5.9 is currently pinned to taxcalc >=0.9.0. Be sure to check the pkg itself by downloading one of the .bz2 files from https://anaconda.org/ospc/ogusa/files, unzipping it and looking at the meta.yaml of that unzipped dir's recipe dir. Note the meta.yaml of the repo is not what you install exactly (the meta.yaml of the repo is programmatically modified to do the pinning I quoted and we arrived at that way of doing things after having confusion related to no pinning or inconsistent pinning).

[martinholmer]

@PeterDSteinberg said:

I don't know what you mean by "breaks" - can't install it at all or has import problems after installation or fails later on in the middle of a calculation? It's hard to attack these problems without tracebacks and details about how it is being installed. For example, is the dev label being used, is it a new conda environment we are sure is free of prior installation-related complications, etc.?

@PeterDSteinberg, You didn't read my question carefully. The work "breaks" is from the @jdebacker comment on OG-USA release 0.5.9. So, I'm asking him what he means by "breaks". I have no idea what he means, which is why I asked the question. Do you understand the conversation now?

@MattHJensen

[martinholmer]

@PeterDSteinberg said:

OG-USA 0.5.9 is currently pinned to taxcalc >=0.9.0. Be sure to check the pkg itself by downloading one of the .bz2 files from https://anaconda.org/ospc/ogusa/files, unzipping it and looking at the meta.yaml of that unzipped dir's recipe dir. Note the meta.yaml of the repo is not what you install exactly (the meta.yaml of the repo is programmatically modified to do the pinning I quoted and we arrived at that way of doing things after having confusion related to no pinning or inconsistent pinning).

This is chaos; there is nothing in the conda.recipe that says OG-USA is pinned to taxcalc >= 0.9.0
Why are you making all this so complicated? If there are reasons that require the complexity of having the repo [be] programmatically modified to do the pinning, then please spell out those reasons.

@MattHJensen @jbcrail

[PeterDSteinberg]

It was quite difficult before when there was no pinning at all or pinning that was done manually. We agree there needs to be pinning of some sort and the most reliable way, IMHO, to do that is via programmatic edits to the yaml. Alternatively we could theoretically go back to how it was before, leaving the pkg maintainers to edit the meta.yaml, but then the OG-USA / B-Tax maintainers have to remember to look at the conda.recipe and ensure it is always up-to-date with Tax-Calculator version(s) they expect the pkgs to go with.

The server deployments have become more predictable and less expensive in human and AWS time due to the changes in policybrain-builder like this programmatic pinning. It's of course not my decision how we proceed, but we should stick with the way we are doing it now or be aware if we are switching it back to how it was that the old method had its own flavor of difficulty. I think some of the difficulty relates to the fact that we are a dispersed team and not seeing immediately who is in chaos mode - some decisions tend to push the confusion towards the pkg maintainers while others push confusion towards the server deployment engineers.

[martinholmer]

@PeterDSteinberg said in discussion of OG-USA issue #306:

It was quite difficult before when there was no pinning at all or pinning that was done manually. We agree there needs to be pinning of some sort and the most reliable way, IMHO, to do that is via programmatic edits to the yaml. Alternatively we could theoretically go back to how it was before, leaving the pkg maintainers to edit the meta.yaml, but then the OG-USA / B-Tax maintainers have to remember to look at the conda.recipe and ensure it is always up-to-date with Tax-Calculator version(s) they expect the pkgs to go with.

Yes, we do agree that OG-USA and B-Tax need to pin to the latest version of the taxcalc package that the developers know works with their model. But we disagree on the best way to do that.

Above you talk about "go[ing] back to how it was before, leaving the pkg maintainers to edit the meta.yaml". But when I look at the histories of the meta.yaml files in the OG-USA and B-Tax repos, I see that taxcalc has never been pinned manually. So, it would seem that the approach more consistent with the Release Process described in webapp-public issue 560 --- that is, having "the OG-USA / B-Tax maintainers have to remember to look at the conda.recipe and ensure it is always up-to-date with Tax-Calculator version(s) they expect the pkgs to go with --- has never been tried. As developers in charge of a model that depends on the taxcalc pacakge, wouldn't their routine testing make it clear what version of taxcalc their model works with? Their comments on the releases indicate that they do know which version of taxcalc their model works with.

@PeterDSteinberg also said:

The server deployments have become more predictable and less expensive in human and AWS time due to the changes in policybrain-builder like this programmatic pinning.

Maybe the policybrain-builder script has helped in other ways, but there is no basis to say "programmatic pinning" has helped. I say this because "manual pinning" has never been tried. So, I don't see how you can say "programmatic pinning" is better than "manual pinning" when manual pinning was never the approach. Certainly, no pinning will not work because the latest version of taxcalc will be made available to the model, which might not be ready for the latest taxcalc version.

@MattHJensen @jdebacker @rickecon @jbcrail

[PeterDSteinberg]

@martinholmer Part of the confusion is that the manual pinning of the past, when it was done, was done outside of version control as part of custom edits that were part of the old build process. There were also times when there was no pinning. I don't recall the history exactly (I'm referring to pre-policybrain-builder times in general, ca 4 to 8 months ago), but the confusion I'm discussing was one of the major motivators for making policybrain-builder work the way it does (requiring building against a known Tax-Calculator tag). We also use to have a ./distribution folder in OG-USA and nearly identical one in Tax-Calculator, where each of them needed some minor edits in order to run. Standardizing that manual editing process has helped us and also helped us avoid making a third nearly identical ./distribution folder (B-Tax).

Pinning-related confusion is a common thing in complicated projects like this one. We have a demo on Wednesday about conda packaging and environments where we'll talk about the status of policybrain-builder, Jenkins, and other tools. That would be a good time to discuss further how we should proceed. My opinion at this point is that policybrain-builder needs to have its issues addressed ( @jbcrail ), but it is easy to do a one-click release with Jenkins now. About 6 months ago, a release process took about an hour of human time.

[jdebacker]

@PeterDSteinberg Thanks for the background the the issues of pinning to specific packages for the conda recipe. I look forward to the tutorial later this week.

@martinholmer To explain my comment in the release tag, my intent was to remind me, Peter, and other OG-USA users that while I created a new tag, I did so with a OG-USA that interacted with TC 0.9.0 in the way it should, but broke in other areas due to incorrect package dependencies in the environment.yml for OG-USA. Until a few days ago, I didn't know what packages/versions were appropriate for OG-USA because of updates to some of it's dependencies. Thanks to @MattHJensen, I think I've got that figured out and have opened PR #305 to set an environment that should work. We have relatively poor testing for OG-USA, so I think our plan, now that we can run OG-USA again, is to write better tests, then try to get OG-USA working with an environment that is consistent with what TC and B-Tax run in.

[MattHJensen]

@jdebacker, said:

we have relatively poor testing for OG-USA, so I think our plan, now that we can run OG-USA again, is to write better tests, then try to get OG-USA working with an environment that is consistent with what TC and B-Tax run in.

That sounds great.

[MattHJensen]

In regard to where and how this projects' dependencies should be pinned, I'll quote a comment I just left over at B-Tax:

I think the contributors to this project should adjust the pins via PRs to this project.

This places the ultimate responsibility for managing dependencies, including testing, on the core maintainer of this project rather than the maintainers of policybrain-builder.

[martinholmer]

@jdebacker said:

To explain my comment in the release tag, my intent was to remind me, Peter, and other OG-USA users that while I created a new tag, I did so with a OG-USA that interacted with TC 0.9.0 in the way it should, but broke in other areas due to incorrect package dependencies in the environment.yml for OG-USA. Until a few days ago, I didn't know what packages/versions were appropriate for OG-USA because of updates to some of it's dependencies. Thanks to @MattHJensen, I think I've got that figured out and have opened PR #305 to set an environment that should work. We have relatively poor testing for OG-USA, so I think our plan, now that we can run OG-USA again, is to write better tests, then try to get OG-USA working with an environment that is consistent with what TC and B-Tax run in.

@jdebacker, Thanks for the explanation. Glad you're making progress on these complex matters.

[martinholmer]

My original question about OG-USA release 0.5.9 has been answered, so I'm closing issue #306.