Council Area Profiles

Rmarkdown to generate demographic profiles for each Scottish Council area.

How it works

For each council area we create: data, plots, tables, and text. These elements are then knitted into an HTML document by an RMarkdown file. The whole process is controlled by a single script that loops over each council area.

How to update

1. Save a copy of the most recent files to a local drive (RMarkdown will knit faster in a local drive)
2. Update the data file (including data and last/next updates)
3. Check you’re using the latest dataset and only the updated datasets have changed
4. Check all the 'next update' dates are still in the future
5. Update links to source tables
6. Run the script that loops over each council area (this normally takes around 20 mins)
7. Sense check the HTML document
8. Send a sample of HTML documents to the relevant Statistician for checking
9. Send all HTML documents to the web team to upload to the website
10. Set a reminder for the next update
11. After publication upload changes to GitHub
12. Delete old local copies of the code

Roles and responsibilities

Branches:

• Provide updated data up to 1 week after any relevant publications
• Let central team know when data has been updated
• Check all content in HTML files is correct (including: data, links, text, next update estimates)
• Correct any issues flagged by the QA scripts. This includes minor issues like "and" vs "&" to retain data provenance.

Statistical Promotion and Analysis:

• Run code and share with branches for QA within 1 week of branches updating any datasets
• Make any changes required by branches and provide web team with HTML files

Statistics Dissemination:

• Upload HTML files within 1 week of them being available

Who owns each tab

Household Estimates and Projections:

• Household estimates
• Household projections
• Dwellings
• Dwellings by type
• Dwellings by council tax band

Vital Events:

• Births by sex
• Standardised birth rates
• Births by age of mother
• Fertility rates
• Deaths by sex
• Standardised death rates
• Deaths by sex by age
• Leading causes of death
• Life expectancy
• Marriages
• Civil partnerships

Population and Migration Statistics:

• Population estimates
• Population projections
• Nature of population change
• Migration
• Net Migration
• Met migration rates

Room for improvement

In rough order of importance/urgency:

• Validate data before preparing tables (e.g. is every council area included? Should Scotland be included? Are all combinations of all variables included? Are values sensible e.g. age < 120?)
• Simplify the code. For example:
  • Replace deeply nested ifelse with case_when or switch statements
  • Use functions and apply DRY
  • Use existing utilities
• Reduce the time it takes to run
  • This may be because some datasets are very large
  • A solution could be to create aggregate versions of a single dataset
• Automate horizontal axis labels so the final label is for the final datapoint (same for the first)
• Improve/shorten variable names
• Improve comments (e.g. why not what)
• Nest/tidy datasets so the environment is more managable
• Fix Y axis on fertility plot
  • Plot function
  • Fertility plot code
  • Fertility rate data

Licence

This repository is available under the Open Government Licence v3.0.