Error running the example script

[jdebacker]

@jpycroft reports:

Then I thought I’d try to run the example “python run_oguk.py” in the “examples/” folder. This generated over 5000 lines of errors. Many errors reference matplotlib (I checked that my matplotlib is up-to-date, of course). I've also attached that output below.

See Issue #28 for the above referenced txt files.

[jdebacker]

The matplotlib lines are warnings, I believe. It maybe that the OG-Core style file needs updating, or it could be an issue with trying to import this file from a URL as is done in both OG-USA and OG-UK (see here).

The real issue, and what is breaking execution, is

Number of workers =  4
Using baseline tax parameters from  C:\Users\jonat\repos\OG-UK\examples\OG-UK-Example\OUTPUT_BASELINE\TxFuncEst_baseline.pkl
distributed.worker - WARNING -  Compute Failed
Function:  get_calculator_output
args:      (True, 2018, None, 'cps')
kwargs:    {}
Exception: KeyError("Unable to open object (object 'P_person_id' doesn't exist)")

Traceback (most recent call last):
  File "run_oguk.py", line 176, in <module>
    main()
  File "run_oguk.py", line 60, in main
    c = Calibration(p, estimate_tax_functions=True, client=client)
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 47, in __init__
    tax_func_path=tax_func_path,
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 126, in get_tax_function_parameters
    num_workers=num_workers,
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 138, in get_data
    results = client.gather(futures)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1992, in gather
    asynchronous=asynchronous,
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 833, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 324, in f
    result[0] = yield future
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 44, in get_calculator_output
    sim = Microsimulation(year=year)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 243, in __init__
    self.load_dataset(dataset, self.year)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 357, in load_dataset
    "person": np.array(data["P_person_id"][year]),
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\h5py\_hl\group.py", line 305, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'P_person_id' doesn't exist)"
distributed.worker - WARNING - Heartbeat to scheduler failed

(oguk-dev) C:\Users\jonat\repos\OG-UK\examples>

I've seen this issue with object 'P_person_id' doesn't exist before, but I can't recall exactly how it was resolved. It was some issue with the data used for OpenFisca-UK, I believe. Let me see if I can recreate.

[jdebacker]

@jpycroft I was able to generate this same error. I had thought I ran this example before, but not sure.

PR #27 maybe helpful here (though it needs to be updated with the master).

Also, it would be good to learn more about OpenFisca-UK. I'm still not totally sure how to do many things with the model and don't fully understand how the data is loaded and hierarchical data structures handled. It might be good for all of us working with OG-UK to learn more about the microsimulation model.

[jpycroft]

@jdebacker @nikhilwoodruff @rickecon

Continuing to try to understand why I getting the litany of errors with run_oguk.py.

I realise now that I'm not even getting past the import commands at the beginning of file. If I run to here:

import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
from oguk.calibrate import Calibration
print('stop after oguk.calibrate')
sys.exit()

I already get a load of errors: see attached "run_oguk_stop_after_importCalibrate_11oct2021.txt"

Going to the next import command, and commenting out the previous import command, i.e.:

import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
# from oguk.calibrate import Calibration
from ogcore import output_plots as op
print('stop after ogcore - output plots; Calibration off')
sys.exit()

I again get a load of errors: see attached "run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt"

The same happens for the other "from ogcore" imports.

run_oguk_stop_after_importCalibrate_11oct2021.txt
run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt

[jdebacker]

@jpycroft All those were warnings. Your set of commands did execute (see the stop after ogcore - output plots; Calibration off printed at the end).

But those messages are annoying. I just merged a PR to OG-Core and a PR to OG-UK that should fix this (they completely remove any imports of matplotlib style files from scripts used in the two packages). Install new versions of each and try that above again. That warnings should disappear.

[jpycroft]

Thanks @jdebacker!

I updated my branch. It hard to tell if there are fewer warnings. There are certainly still many (attached).

In any case, if it's only warnings, then perhaps the only important part of the output is the last lines:

usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

I'll have a look at this later on.

run_oguk_after-poss-removing-some-warnings_12oct2021.txt
.

[jdebacker]

@jpycroft Did you reinstall the ogcore package? Did you rebuild the oguk package after updating your branch from the upstream main branch?

[jpycroft]

@jdebacker Thanks again!

I hadn't updated my OG-Core. I updated that and tried to install its environment. This only partially worked, ending with the following error message (full output attached):

Pip subprocess error:
ERROR: Could not install packages due to an OSError: [WinError 5] Acceso denegado: 'C:\\Users\\jonat\\AppData\\Local\\Temp\\pip-uninstall-lahmt6kq\\sphinx-apidoc.exe'
Consider using the `--user` option or check the permissions.

failed

CondaEnvException: Pip failed

In any case, I updated OG-UK again and updated the environment (oguk-dev). Trying to run "run_oguk.py" still results in the litany of warnings as before, culminating in the same error as above (full output attached):

usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

og-core_try_to_build_environment_13oct2021.txt

run_oguk_after-updating_13oct2021.txt

[nikhilwoodruff]

Great to see it (at least partially) working @jpycroft - this one I can answer: I while ago I wrote this format for run_oguk.py to take as argument a reform object in a Python file - it should work with python run_oguk.py small_ubi_reform.ubi_reform (pointing to the ubi_reform object in small_ubi_reform.py). I can take it out if it's causing inconvenience.

[jdebacker]

@jpycroft, Nikhil pointed out the issue with your attempted run of the example script.

I think the matplotllib warnings are still there because OG-Core was not updated because your env failed to updated.

I'd remove the environment with: conda remove --name oguk-dev --all

Then create it anew with conda env create

[jdebacker]

@rickecon because he may have seen the same sphinx install error that Pycroft is seeing recently. Suggests perhaps removing those dependencies that just help with documentation.

[jpycroft]

Thanks for the ideas @nikhilwoodruff @jdebacker.

Running the python run_oguk.py small_ubi_reform.ubi_reform (having reinstalled the environment) I get the same litany of errors, but now I'm ending on the person_id error:

KeyError: "Unable to open object (object 'person_id' doesn't exist)"

Output attached.
Best, Jon

run_oguk_small_ubi_reform_23oct2021.txt

[jpycroft]

@rickecon, @jdebacker, @nikhilwoodruff,

Exciting news: I’m getting different error messages :-) Seriously, that is exciting!

Having updated to the latest environment.yml and doing “conda update conda”, the litany of hundreds of error messages is gone. I’ve worked my way through a number of error messages and seem now to be stuck on accessing the synthetic data set for OpenFiscaUK.

I’ll follow this up below with two more posts on:

1. Adjustments I’ve already made to get past errors &
2. Notes on error messages regarding the default / synthetic data set

[nikhilwoodruff]

Amazing - admittedly it's been a while since I last used the synthetic dataset, happy to debug anything else you get. Or, I don't suppose you can get access directly to the FRS by any means?

[jpycroft]

Hi @nikhilwoodruff,
Yes, going for the full FRS might be the best solution.
I'll look into how to download. It seems that I'll need to agree to an End User Licence first ...

[jpycroft]

With reference to the thread above, #31 (comment)
I'm trying to run: python run_og_uk.py

Adjustments I’ve already made to get past errors:

1. change line in run_og_uk.py:
   from og_uk_calibrate.calibrate import Calibration
   --> from oguk.calibrate import Calibration

2. Copy two files into the OG-UK folder:
   a. ogusa
   b. taxcalc (pip install taxcalc did not work. seemingly many errors: "Preparing metadata (setup.py) ... error" ... "Command errored out with exit status 1"; then tries again with earlier version (e.g. 0.6.2.) and same message appears; also tried pip install taxcalc==0.4 / 0.4.1 / etc), Instead, fork & git clone Tax-Calculator. Then cut & paste "taxcalc" folder into OG-UK

3. Re-install a few packages, which were missing because I had reinstalled Anaconda:
   pip install numba
   pip install eurostat
   pip install xlsxwriter

4. Remove command to put OG-USA version in parameters to save for reference
   As ogusa is no longer part of the environment installation, the line in parameters.py:
   self.ogusa_version = pkg_resources.get_distribution("ogusa").version
   Throws an error:

  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pkg_resources\__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ogusa' distribution was not found and is required by the application

As it appears to be non-essential, for now I simply edited it out.

5. The next error is a failure to find the default / synthetic data set; see next comment.

@jdebacker @rickecon @nikhilwoodruff

[jpycroft]

With reference to the thread above, #31 (comment) & #31 (comment)
I'm trying to run: python run_og_uk.py

Notes on error messages regarding the default / synthetic data set:

1. Error when running OG-UK:
   FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = 'C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk_data\microdata\openfisca_uk\frs_was_imp_None.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
   Checking the file location, all folders with \microdata\ are empty.

2. Retrying to run OpenFiscaUK in isolation.
   This was a suggestion from Nikhil that had previously worked, see OG-UK set up #21 (comment) & OG-UK set up #21 (comment).
   Trying to do the same now, gives the error:

In [2]: sim = Microsimulation() # using the synthetic dataset
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-eef7f274daa0> in <module>
----> 1 sim = Microsimulation() # using the synthetic dataset

~\anaconda3\envs\test\lib\site-packages\openfisca_tools\microsimulation.py in __init__(self, reform, dataset, year)
     40             self.dataset = dataset
     41         if year is None:
---> 42             self.year = self.default_year or max(self.dataset.years)
     43         else:
     44             self.year = year

ValueError: max() arg is an empty sequence

This is a result of their being no default data set.

Possible solution
@nikhilwoodruff has already suggested (#31 (comment)) that, rather than worry about the synthetic data set, it may be more efficient to get access directly to the Family Resources Survey, which is what is needed to run properly anyway. I'm going ahead with the application for access now.

@jdebacker @rickecon

[nikhilwoodruff]

@jpycroft the full FRS would be ideal - good luck - but I wonder if the synthetic dataset works if you do this?

openfisca-uk-setup --set-default synth_frs

[jpycroft]

Hi @nikhilwoodruff,
Thanks for the idea. When I do the openfisca-uk-setup --set-default synth_frs command, I get a different error, and I get it one command earlier in response to from openfisca_uk import Microsimulation: basically it complains that ModuleNotFoundError: No module named 'family_resources_survey'
Here's how I entered it, and the output below.
Best, Jon

(test) C:\Users\jonat\repos\openfisca-uk>openfisca-uk-setup --set-default synth_frs
Just setting default dataset, skipping questions.

(test) C:\Users\jonat\repos\openfisca-uk>ipython
Python 3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from openfisca_uk import Microsimulation
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-5d103e1b8098> in <module>
----> 1 from openfisca_uk import Microsimulation

~\repos\openfisca-uk\openfisca_uk\__init__.py in <module>
      4 from openfisca_core.taxbenefitsystems import TaxBenefitSystem
      5 from openfisca_uk import entities
----> 6 from openfisca_uk.microdata.simulation import Microsimulation
      7 from openfisca_uk.tools.simulation import IndividualSim, PopulationSim
      8 from openfisca_uk.reforms.presets.modelling import (

~\repos\openfisca-uk\openfisca_uk\microdata\simulation.py in <module>
      7 from openfisca_uk.entities import *
      8 from openfisca_core.simulation_builder import SimulationBuilder
----> 9 from openfisca_uk.microdata.frs.dataset import FRSDataset
     10 from openfisca_uk.microdata.spi.dataset import SPIDataset
     11 from openfisca_uk.microdata.frs.config import from_FRS

~\repos\openfisca-uk\openfisca_uk\microdata\frs\dataset.py in <module>
      1 import numpy as np
      2 import pandas as pd
----> 3 from family_resources_survey import FRS
      4 from pathlib import Path
      5 import shutil

ModuleNotFoundError: No module named 'family_resources_survey'

[jdebacker]

@jpycroft writes:

Adjustments I’ve already made to get past errors:

and then lists a bunch of changes he's made to source files to get things to work on his branch. But all of these changes are not relevant for the current versions of the sources files in the main branch.

@jpycroft I would suggest that you sync the branch you are working on with the upstream/main.

[nikhilwoodruff]

Thanks @jpycroft and @jdebacker - yes that's right. The code there is several months out of date. I think you shouldn't even need the repo cloned, but if you want to keep it, I'd suggest deleting it and reinstalling it.

[jdebacker]

Also, when adding packages dependencies (like taxcalc or ogusa (neither of which should be necessary for OG-UK) or openfisca_uk), it would be best to install those as packages (via pip or conda-forge). Second best (or if you need to modify those packages) is to build the package on your machine via the source code (just be sure it's at the state you want it to be at).

The option you should avoid at all costs is to copy over all the source files and place them in a directory inside another project and call them as if they are just modules which is what is outlined above in the comment by @jpycroft that says:

Copy two files into the OG-UK folder:
a. ogusa
b. taxcalc (pip install taxcalc did not work. seemingly many errors:

This is dangerous because you will not be sure you have all the dependencies of the package, it will be difficult to version control, and you are not guaranteed things run.

FYI, current versions of taxcalc are no longer released on pip and need to be installed from conda-forge (conda install -c conda-forge taxcalc).

[jpycroft]

Hi @jdebacker, @nikhilwoodruff,

Let me first say about the sync-ing. Doing a upstream/main sync was the first thing I did. I assumed that was why I get different results than before. I was doing this:

(base) C:\Users\jonat\repos\OG-UK>git fetch upstream
(base) C:\Users\jonat\repos\OG-UK>git merge upstream/main
Already up to date.

OK, maybe this is the problem:

(base) C:\Users\jonat\repos\OG-UK>git remote show upstream
* remote upstream
  Fetch URL: https://github.com/PSLmodels/OG-UK-Calibration.git
  Push  URL: https://github.com/PSLmodels/OG-UK-Calibration.git
  HEAD branch: main
  Remote branches:
    gh-pages tracked
    main     tracked
  Local ref configured for 'git push':
    main pushes to main (fast-forwardable)

My upstream was still linked to the OG-UK-Calibration. Was that it? (If so, I still find that confusing for other reasons.) In any case, I'll do a delete and reinstall as you suggest Nikhil.

[jdebacker]

@jpycroft Ah yes - the repo name change. That could be it. You'll want to reconfigure your remote URLs. Do that with a:

git remote set-url upstream https://github.com/PSLmodels/OG-UK.git

and, since your fork has this name also:

git remote set-url origin https://github.com/jpycroft/OG-UK.git

[jpycroft]

@jdebacker @nikhilwoodruff

Thanks also for highlighting the conda-forge solution. I'll remember that if needed in future. (I should have clarified perhaps, the copying across was only ever intended as a stop gap to try and get moving at all.) Perhaps with the right version, it won't be required here.

[nikhilwoodruff]

Sounds good - though I did think that GitHub redirects name changes, so that actually probably should've worked. But not really a problem anymore

[jpycroft]

@jdebacker @nikhilwoodruff

I removed and re-cloned OG-UK. I also removed and reinstalled the environment again. Now everything seems to be properly up-to-date.

The error I'm hitting now is that I'm missing the 'reform' argument:

(oguk-dev) C:\Users\jonat\repos\OG-UK\examples>python run_oguk.py
Using synthetic dataset.
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

The reform ought to be taken from reform=get_default_reform(), which is a simple UBI reform. It's not clear to me why this isn't being picked up.
Also I'm not clear what line 228 does
reform = getattr(__import__(python_module), object_name)
Any ideas would be welcome. Thanks!

[nikhilwoodruff]

OK, so that line essentially takes the argument, e.g. small_ubi_reform.ubi_reform, separates it into the module path (small_ubi_reform) and object name ubi_reform, and imports ubi_reform from its declaration in small_ubi_reform.py.

I wonder if either of the following works:

• main(reform) -> main() on line 228 - er, actually this is line 228 on the repo rather than the one you quoted, perhaps you might be out of date?
• Running python run_oguk.py small_ubi_reform.ubi_reform from OG-UK\

[jpycroft]

Hi @nikhilwoodruff,

Thanks for the suggestions and clarification.

Switching from main(reform) -> main(), it still wants a reform:
(btw, I'd added a print above, which explains the line change.)

(oguk-dev) C:\Users\jonat\repos\OG-UK>python examples/run_oguk.py
Using synthetic dataset.
confirm: name does equal __main__
usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

With main() or main(reform), I get this output running with small_ubi_reform.ubi_reform:

(oguk-dev) C:\Users\jonat\repos\OG-UK>python examples/run_oguk.py small_ubi_reform.ubi_reform
Using synthetic dataset.
confirm: name does equal __main__
Using synthetic dataset.
Using synthetic dataset.
Using synthetic dataset.
Using synthetic dataset.
Number of workers =  4
Using baseline tax parameters from  C:\Users\jonat\repos\OG-UK\examples\OG-UK-Example\OUTPUT_BASELINE\TxFuncEst_baseline.pkl
distributed.worker - WARNING -  Compute Failed
Function:  get_calculator_output
args:      (True, 2018, None, 'frs')
kwargs:    {}
Exception: ValueError('max() arg is an empty sequence')

Traceback (most recent call last):
  File "examples/run_oguk.py", line 229, in <module>
    main(reform)
  File "examples/run_oguk.py", line 96, in main
    c = Calibration(p, estimate_tax_functions=True, client=client)
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 41, in __init__
    tax_func_path=tax_func_path,
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 120, in get_tax_function_parameters
    num_workers=num_workers,
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 191, in get_data
    results = client.gather(futures)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1992, in gather
    asynchronous=asynchronous,
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 833, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 324, in f
    result[0] = yield future
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 94, in get_calculator_output
    sim = Microsimulation()
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_tools\microsimulation.py", line 42, in __init__
    self.year = self.default_year or max(self.dataset.years)
ValueError: max() arg is an empty sequence

So, it appears to be recognising the reform. It states that it's going to use the synthetic data set, but fails to find the default data.

@jdebacker @rickecon

[jpycroft]

Hi @nikhilwoodruff,

Back to trying to get the model running with FRS. Finally got round to trying this out over the weekend.

First, OG-UK does begin to run with synthetic data. To run with FRS, you mentioned to
i. pip install openfisca-uk - worked fine for me - then
ii. run openfisca-uk

I'm not clear which command to use to get openfisca-uk to run. I first tried the command suggested by OG-UK:
openfisca-uk-data frs_enhanced download 2019
which gave the error below:

WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...

Do I need to set up the FRS data somewhere, or be logged in to UKdata?
I tried various permutations to the command to run openfisca, but haven't hit on a combination that works.
If you have any thoughts, please share, and hope to catch you later at the meeting.

[nikhilwoodruff]

Thanks @jpycroft - good to know the synth FRS runs. That gcloud error looks strange - I should be able to have a proper look today before the OG meeting, but looks like it's been found by others too.

[nikhilwoodruff]

As per that issue could you try running gcloud auth application-default login?

[jpycroft]

Hi @nikhilwoodruff,

Thanks for the idea! I get this:

(oguk-dev) C:\Users\jonat\repos\OG-UK>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.

Translation: "gcloud is not recognised as an internal or external command, executable programme or batch file."

I pip installed gcloud and then google-cloud (though the pypi docs say that neither is currently maintained).
In any case, the error message stays the same.

[jpycroft]

Hi @nikhilwoodruff,

Thanks for adding this change: https://github.com/PolicyEngine/openfisca-tools/blob/master/setup.py

I did a "pip install openfisca-uk" should update my version to try to incorporate it. Unfortunately, I'm still getting the same output. Is there something else I need to run first?

(oguk-dev) C:\Users\jonat\repos\OG-UK>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.

(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...

[nikhilwoodruff]

Ah thanks @jpycroft - sorry, could you try pip install --upgrade openfisca-uk now? I've added google-cloud-storage as a dependency, so it should get installed (might be worth running pip install google-cloud-storage after, just to be sure, but this should happen automatically).

One of the differences between my local setup and yours is that I have the gcloud CLI installed. But I'm aiming for that to not be required to download the microdata.

[jpycroft]

Hi @nikhilwoodruff,

I get a different output now, which is surely progress!

I entered:

1. pip install --upgrade openfisca-uk
2. pip install google-cloud-storage (though everything was "Requirement already satisfied" as you thought it would be.)

Then openfisca-uk-data frs_enhanced download 2019 gives this:

(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
Traceback (most recent call last):
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\Scripts\openfisca-uk-data.exe\__main__.py", line 4, in <module>
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\__init__.py", line 6, in <module>
    from openfisca_uk.tools.simulation import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 3, in <module>
    from openfisca_uk.data import EnhancedFRS
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\__init__.py", line 1, in <module>
    from openfisca_uk.data.datasets import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\__init__.py", line 1, in <module>
    from openfisca_uk.data.datasets.frs import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\__init__.py", line 1, in <module>
    from .frs import FRS
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\frs.py", line 2, in <module>
    from openfisca_tools.data import PrivateDataset
ModuleNotFoundError: No module named 'openfisca_tools.data'

[nikhilwoodruff]

Great stuff @jpycroft - the good news is that I know exactly what this is, and I think it should be fixed now if you update OpenFisca-UK (for context, this issue was caused by me naively removing MANIFEST.in from the repo without realising what it did: PolicyEngine/policyengine-uk#574).

[jpycroft]

Thanks again @nikhilwoodruff!

I redid the pip install --upgrade openfisca-uk. (I think) everything was "Requirement already satisfied" except for:

Installing collected packages: argparse
Successfully installed argparse-1.4.0

Then the same command as before (I think) still gives the same output. Any other ideas?

(oguk-dev) C:\Users\jonat\repos\OG-UK>openfisca-uk-data frs_enhanced download 2019
Traceback (most recent call last):
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\Scripts\openfisca-uk-data.exe\__main__.py", line 4, in <module>
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\__init__.py", line 6, in <module>
    from openfisca_uk.tools.simulation import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 3, in <module>
    from openfisca_uk.data import EnhancedFRS
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\__init__.py", line 1, in <module>
    from openfisca_uk.data.datasets import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\__init__.py", line 1, in <module>
    from openfisca_uk.data.datasets.frs import (
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\__init__.py", line 1, in <module>
    from .frs import FRS
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\data\datasets\frs\frs.py", line 2, in <module>
    from openfisca_tools.data import PrivateDataset
ModuleNotFoundError: No module named 'openfisca_tools.data'

[nikhilwoodruff]

Sorry for my delay in getting back to this @jpycroft! Maybe we can debug it a bit at the meeting tomorrow? One thought: running the --upgrade flag should result in some message at the end saying "uninstalling openfisca-uk... installed openfisca-uk", did this appear?

[jpycroft]

Hi @nikhilwoodruff,

Thanks again. The command (oguk-dev) C:\Users\jonat\repos>pip install --upgrade openfisca-uk gave a long list of "Requirement already satisfied" and ended with this:

Installing collected packages: argparse, openfisca-uk
  Attempting uninstall: openfisca-uk
    Found existing installation: OpenFisca-UK 0.20.3
    Uninstalling OpenFisca-UK-0.20.3:
      Successfully uninstalled OpenFisca-UK-0.20.3
Successfully installed argparse-1.4.0 openfisca-uk-0.20.4

So yes, it does give the "uninstalling openfisca-uk... installed openfisca-uk" message.

[jpycroft]

Hi @nikhilwoodruff,

Actually, it's a bit disappointing ... it didn't quite solve the issue. Here's what happened.

First, pip installing openfisca-uk within the newly created "openfisca-uk" environment (with Python 3.7.13) did eventually work. Output attached as txt.
pip install openfisca-uk_25apr2022.txt

Then the command to download 2019 frs data gave the following:

(openfisca-uk) C:\Users\jonat\repos>openfisca-uk-data enhanced_frs download 2019
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
INFO:root:Could not automatically authenticate with Google Cloud, prompting login...


Encountered an error:
Traceback (most recent call last):
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 19, in _get_storage_bucket
    client = storage.Client()
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\storage\client.py", line 165, in __init__
    _http=_http,
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 320, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 268, in __init__
    project = self._determine_default(project)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\client\__init__.py", line 287, in _determine_default
    return _determine_default_project(project)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\cloud\_helpers\__init__.py", line 152, in _determine_default_project
    _, project = google.auth.default()
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\google\auth\_default.py", line 579, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\Scripts\openfisca-uk-data.exe\__main__.py", line 7, in <module>
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_uk\data\cli.py", line 6, in cli
    openfisca_data_cli(DATASETS)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\cli.py", line 33, in openfisca_data_cli
    raise e
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\cli.py", line 26, in openfisca_data_cli
    result = target(*args.args)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\dataset.py", line 69, in wrapper
    return fn(*args, **kwargs)
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 55, in download
    bucket = self._get_storage_bucket()
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\site-packages\openfisca_tools\data\private.py", line 28, in _get_storage_bucket
    stderr=subprocess.DEVNULL,
  File "C:\Users\jonat\anaconda3\envs\openfisca-uk\lib\subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gcloud auth application-default login']' returned non-zero exit status 1.

PS: I retried the gcloud auth, but same error as before:

(openfisca-uk) C:\Users\jonat\repos>gcloud auth application-default login
"gcloud" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.

[nikhilwoodruff]

Ah, well thanks for getting through the installation: I think it's still a step in the right direction. The error makes sense to me: it's because the gcloud CLI is not installed. I've tried to avoid it by trying to find ways to not require its installation, but at this point it seems inevitable unfortunately. So, could you please install the tool on your machine and try it again? Thanks.