[WIP] Creating UK demographics.py file

[jpycroft]

This creates a UK demographics.py file.

cc: @jdebacker @rickecon

[rickecon]

@jpycroft . Will you please do the following on your demog branch to update it? Since you created your branch for this PR, a couple of PRs have been merged in. I am having trouble pulling yours to add a PR to your branch because it is out of sync with the PSLmodels repo.

1. In your local OG-UK-Calibration repository in the terminal, change to your demog branch using git checkout demog
2. Make sure that any files that have been modified, added, or deleted since your last commit (see this by typing git status) are committed and pushed to your remote fork.
3. Fetch the updated version of the PSLmodels/OG-UK-Calibration repository by typing git fetch upstream where upstream is the remote for the PSLmodels/OG-UK-Calibration repository.
   • If you haven't set the upstream remote, you can do this from the terminal in your local fork repository by typing git remote add upstream https://github.com/PSLmodels/OG-UK-Calibration.git.
4. Merge the changes from the PSLmodels main branch into your local demog branch by typing git merge upstream/main
   • This may open a screen that involves some decisions.
   • The only merge conflicts you might have will come from the folder name change from oguk_calibration to og_uk_calibrate. I don't know what git will do if you just accept the changes from the upstream remote. My guess is that it will create two folders, (i) og_uk_calibrate with the files from the new updates and (ii) oguk_calibration containing the changes you made.
   • You might want to save a copy of your demographics.py and EurostatDemographics.py just in case.

Contact me if you run into any problems or have any questions. Once you do this, I can pull your branch and submit my PR to your branch.

[rickecon]

@jpycroft. I was able to successfully able to merge your current branch demog into a branch in my fork. Instead of following the git checkout -b ... and git pull [fork/branch] paradigm, I did the following.

1. I created a new branch called jpycroft-demog.
2. Then I created a git remote for your fork of the repository.
3. Then I just did a git merge from your branch to mine.
4. The merge conflict was the directory name og_uk_calibrate rather than your oguk_calibration. This is a nonstandard merge conflict, but ended up being easy to solve.
5. Git automatically recognized that the problem was the directory name, which showed when I typed git status.
6. I just had to git add, git commit, and git push the directory name change.

Now I should be able to submit my PR to your branch. I am going to try to have that done in the next hour.

[jpycroft]

@rickecon
Thanks Rick, this is great! I've now merged your pull request with my demog branch. Jon

[jpycroft]

Hi @jdebacker @rickecon,

To update you on my work on the demographics file.

The method for downloading Eurostat data no longer works. I've spent the morning investigating, and even the tests for the get_sdmx_data_df commands listed in the Eurostat Python Package no longer work either. Other parts of the package do work, like get_data_df, and it looks like I'll have to rewrite the code to use those instead.

For the record, running the example given for Eurostat Python Package:

import eurostat
StartPeriod = 2007
EndPeriod = 2008
filter_pars = {'FREQ': ['A',], 'PRCCODE': ['08111250','08111150']}
df = eurostat.get_sdmx_data_df('DS-066341', StartPeriod, EndPeriod, filter_pars, flags = True, verbose=True)

now gives the following output, which suggests that the data does not exist (I've tried numerous DSD's that did previously work):

(oguk-dev) C:\Users\jonat\repos\OG-UK\oguk>python check_sdmx.py
Server Error 500: DSD_DS-066341 not found in the Eurostat server
Traceback (most recent call last):
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pandasdmx\model.py", line 791, in __call__
    rc = getattr(self._reader.message, rc_name)
AttributeError: 'StructureMessage' object has no attribute 'datastructure'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "check_sdmx.py", line 72, in <module>
    df = eurostat.get_sdmx_data_df('DS-066341', StartPeriod, EndPeriod, filter_pars, flags = True, verbose=True)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\eurostat\eurostat.py", line 304, in get_sdmx_data_df
    d = get_sdmx_data(code, StartPeriod, EndPeriod, filter_pars, flags, verbose)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\eurostat\_decorators.py", line 61, in wrapper
    res = func(estat, *args, **kwargs)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\eurostat\eurostat.py", line 269, in get_sdmx_data
    resp = __pandasdmx_get_data(estat, code, key=dict(c), params={'startPeriod': str(StartPeriod), 'endPeriod': str(EndPeriod)})
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\eurostat\_decorators.py", line 19, in wrapper
    resp = func(*args, **kwargs)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\eurostat\eurostat.py", line 371, in __pandasdmx_get_data
    resp = estat.data(code, key=key, params=params)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pandasdmx\api.py", line 309, in get
    request=True, target_only=False)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pandasdmx\model.py", line 806, in __call__
    agency=self.agency_id, **kwargs)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pandasdmx\api.py", line 363, in get
    raise SDMXException('Server error:', status_code, url)
pandasdmx.api.SDMXException: ('Server error:', 500, 'http://ec.europa.eu/eurostat/SDMX/diss-web/rest/datastructure/ESTAT/DSD_DS-066341/latest?references=all')

Whereas using get_data_df:

df = eurostat.get_data_df('demo_pjan')
print(df)

does work fine:

(oguk-dev) C:\Users\jonat\repos\OG-UK\oguk>python check_sdmx.py
      unit     age sex geo\time  2021       2020  ...       1965       1964       1963       1962       1961       1960
0       NR   TOTAL   F       AD   NaN        NaN  ...        NaN        NaN        NaN        NaN        NaN        NaN
1       NR   TOTAL   F       AL   NaN  1425342.0  ...        NaN        NaN        NaN        NaN        NaN        NaN
2       NR   TOTAL   F       AM   NaN  1562689.0  ...        NaN        NaN        NaN        NaN        NaN        NaN
3       NR   TOTAL   F       AT   NaN  4522292.0  ...  3857760.0  3836415.0  3814191.0  3794130.0  3773097.0  3757167.0
4       NR   TOTAL   F       AZ   NaN  5039100.0  ...        NaN        NaN        NaN        NaN        NaN        NaN
...    ...     ...  ..      ...   ...        ...  ...        ...        ...        ...        ...        ...        ...
17524   NR  Y_OPEN   T       SM   NaN        NaN  ...        NaN        NaN        NaN        NaN        NaN        NaN
17525   NR  Y_OPEN   T       TR   NaN     5567.0  ...    70967.0    69029.0    67153.0    65335.0    63574.0    61869.0
17526   NR  Y_OPEN   T       UA   NaN    11193.0  ...        NaN        NaN        NaN        NaN        NaN        NaN
17527   NR  Y_OPEN   T       UK   NaN        NaN  ...   400000.0   400000.0   400000.0   400000.0   300000.0   300000.0
17528   NR  Y_OPEN   T       XK   NaN      399.0  ...        NaN        NaN        NaN        NaN        NaN        NaN

[jpycroft]

The pull request has been replaced by #36