New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Councillors data easy to update for volunteers #98

Open
equivalentideas opened this Issue Feb 20, 2017 · 11 comments

Comments

Projects
None yet
6 participants
@equivalentideas
Member

equivalentideas commented Feb 20, 2017

Currently to update the data in here you have to update a spreadsheet carefully and then run a rake task locally, and use git to commit it. This is a lot of assumed knowledge.

How might we make updating this data accessible to all the people who might want to contribute?

Also, who are these people? What are their needs and requirements for access?

@Mianto

This comment has been minimized.

Show comment
Hide comment
@Mianto

Mianto Mar 28, 2017

@equivalentideas I would like to work on this project as part of Outreachy Internship. Regarding this project Also, who are these people? What are their needs and requirements for access? What is implied by this?

Mianto commented Mar 28, 2017

@equivalentideas I would like to work on this project as part of Outreachy Internship. Regarding this project Also, who are these people? What are their needs and requirements for access? What is implied by this?

@karinamachado

This comment has been minimized.

Show comment
Hide comment
@karinamachado

karinamachado Apr 3, 2017

Hi, I would like understand how it works this project idea? Thanks.

Hi, I would like understand how it works this project idea? Thanks.

@equivalentideas

This comment has been minimized.

Show comment
Hide comment
@equivalentideas

equivalentideas Apr 4, 2017

Member

Hi @Mianto and @karinamachado . We'll be working on this project with our Outreachy intern (if that's the project they choose).

You can read about how to apply for the internship here: https://www.openaustraliafoundation.org.au/2017/03/24/join-us-for-a-3-month-paid-full-time-internship-and-start-transforming-our-democracy/

Also, who are these people? What are their needs and requirements for access? What is implied by this?

@Mianto Good question :) The implication is that we don't know what the best solution for solving this problem is yet, and that we'd need to answer these questions at the beginning of the project.

Member

equivalentideas commented Apr 4, 2017

Hi @Mianto and @karinamachado . We'll be working on this project with our Outreachy intern (if that's the project they choose).

You can read about how to apply for the internship here: https://www.openaustraliafoundation.org.au/2017/03/24/join-us-for-a-3-month-paid-full-time-internship-and-start-transforming-our-democracy/

Also, who are these people? What are their needs and requirements for access? What is implied by this?

@Mianto Good question :) The implication is that we don't know what the best solution for solving this problem is yet, and that we'd need to answer these questions at the beginning of the project.

@hisayohorie

This comment has been minimized.

Show comment
Hide comment
@hisayohorie

hisayohorie Jun 1, 2017

Collaborator

Today we (@equivalentideas, @henare and @hisayohorie) had a kick-off meeting to solve this issue, and here are the break-down of the issue to make it clear. Also we worked on identifying some of the use cases to come up with potential solutions.

What is the problem?

  • PlanningAlerts needs data of local councillors so people can write to them concerning development proposals. PlanningAlerts covers 150 local councils.
  • The councillors data changes periodically and irregularly and it needs to be updated manually by someone.
  • The OpenAustralia Foundation team does not have the capacity to keep it up to date themselves.
  • The current systems requires a lot of programming knowledge to contribute the data. People don’t even know they can update the data.
  • Very few people are able to make contribution, and so very few people do.

We know we will have solved the problem when...

PlanningAlerts has up-to-date councillor information for every authority it covers. This information is updated by the contribution of volunteers. We have an accessible, easy and (fun) system to update/add councillor information, that acknowledges and cerebrates the work of the volunteers.

Collaborator

hisayohorie commented Jun 1, 2017

Today we (@equivalentideas, @henare and @hisayohorie) had a kick-off meeting to solve this issue, and here are the break-down of the issue to make it clear. Also we worked on identifying some of the use cases to come up with potential solutions.

What is the problem?

  • PlanningAlerts needs data of local councillors so people can write to them concerning development proposals. PlanningAlerts covers 150 local councils.
  • The councillors data changes periodically and irregularly and it needs to be updated manually by someone.
  • The OpenAustralia Foundation team does not have the capacity to keep it up to date themselves.
  • The current systems requires a lot of programming knowledge to contribute the data. People don’t even know they can update the data.
  • Very few people are able to make contribution, and so very few people do.

We know we will have solved the problem when...

PlanningAlerts has up-to-date councillor information for every authority it covers. This information is updated by the contribution of volunteers. We have an accessible, easy and (fun) system to update/add councillor information, that acknowledges and cerebrates the work of the volunteers.

@hisayohorie

This comment has been minimized.

Show comment
Hide comment
@hisayohorie

hisayohorie Jun 1, 2017

Collaborator

Next steps are...

  • Publish design problem and success condition
  • @hisayohorie identify and answer key questions for people who’ve contributed data already to find out what they need in order to start contributing councillor information. @equivalentideas needs to look at the questions and help organize this.
  • @hisayohorie create design principles and publish on issue.
Collaborator

hisayohorie commented Jun 1, 2017

Next steps are...

  • Publish design problem and success condition
  • @hisayohorie identify and answer key questions for people who’ve contributed data already to find out what they need in order to start contributing councillor information. @equivalentideas needs to look at the questions and help organize this.
  • @hisayohorie create design principles and publish on issue.
@equivalentideas

This comment has been minimized.

Show comment
Hide comment
@equivalentideas

equivalentideas Jun 2, 2017

Member

@hisayohorie has listed some great questions that we have for the people who've been updating the councillor information so far. Some of this stuff we can answer already, which is great. It's still important to document that so I'll make a start here:

What devices you use to look into planningalerts?

This will help us learn what devices we need to support in our project.

We know that the current system only works from wide screen devices/“desktop” because you just couldn’t flick between council websites and google sheets and actually accomplish the task on a phone or tablet. I’ve sat with Pip, our main volunteer, and she’s been doing it on a laptop. When @archoo made a big contribution it involved running a scraper and all-sorts, so we can safely assume they weren’t on a phone. (correct us if we're wrong @archoo 😉 )

But—we also want to create something usable by people who use PlanningAlerts more broadly. We can get the device stats from Google Analytics 😄 :

Over the last 7 days sessions are very evenly split between wide and small screen devices, and touch/point and click input (mouse/trackpad) modes:

screen shot 2017-06-02 at 1 02 34 pm

We think people making comments might be people who want to update councillor information so they can write to them. What devices do people successfully making comments use? It's very evenly split as well:

screen shot 2017-06-02 at 1 24 09 pm

I think this means we should be aiming to make something that people can use no-matter what device that bring to the party. That's best practice for a long time anyway. We also design things mobile first any way, mostly because its easier than going the other way, and makes us focus on keeping things simple.

do you use keyboard shortcuts? What other things do you use your computer? How often do you use computer?

I seems to me that these three questions are about trying to work out how proficient contributors are with computers. This is useful to know because it will guide the types of features we can add.

From my experience with them, and @archoo’s feedback below, I think our contributors fall into two groups around their proficiency with data collection tasks:

  1. People who are programmers and are comfortable working with data in formats like JSON and CSV. @archoo is the example of this group so far, but there are a number of people in the OpenAustralia Foundation community that might want to contribute. People in this group explore and develop tooling to help with data collection they do.
  2. People who aren't programmers, but are have a lot of experience using computers to create and edit documents, email etc. in their work. They have lots of experience with collecting and entering data, but they don't use keyboard shortcuts, and aren't familiar with tools that could help or with developing them. Outside of work they're more likely to use smart phones or tablets for web browsing etc. . Pip Brown is a good example of people in their group. There's lots of people potentially in this group among PlanningAlerts users.

That's a simplistic distinction, but useful I think. There's people who are in the middle, and lots of people who are less familiar with data entry too. Two of our working Design Principles are:

  • Strive for universal accessibility
  • Make the process of contribution is obvious and intuitive

So I think we want to make something that people who are less proficient with computers than those two groups ☝️ can use too.

If you would to contribute, does region makes difference? Do you prefer to input data for councillors in your region?

@archoo is in Queensland and contributed data for four councils in that state. Pip Brown started with the councillors that represent them, but then contributed to lots of other areas.

The people who've notified us about out-of-date councillor records via PlanningAlerts have only done it for councillors in their area. They've been commenters concerned that their local councillor options weren't correct; and, councillors and council staff asking us to update it so they receive records correctly.

It seems there's a distinction here between volunteers and people PlanningAlerts

Volunteers like Pip want to contribute beyond just the records for their local area. @archoo was interested in doing it a number of councils in the state they're from.

People using PlanningAlerts, who don't even know this is a project they can contribute to, have only taken interest in their own local area. That could be because the task of updating the records is part of another task for them, like sending or receiving comments correctly (we're had messages from commenters and councillors like this).

How much time do you spend contributing at once?

Pip Brown worked on collecting councillor data in blocks of 2 or so hours. I imaging it took @archoo a few hours to work on scrapers and contribute data for four councils.

Other times people have just done one council at a time. This can still take 20 minutes or so.

What's stopping you from contributing at the moment?

This is a really good and important question @hisayohorie . I think we'll need to ask our current contributors directly about this. It's also the type of question that it's hard for people to answer because they need to describe their own motivations. But, let's give it a go. It might be useful to ask them about other volunteering that they do, and to try and compare why prioritise contributing to some things over others.

Member

equivalentideas commented Jun 2, 2017

@hisayohorie has listed some great questions that we have for the people who've been updating the councillor information so far. Some of this stuff we can answer already, which is great. It's still important to document that so I'll make a start here:

What devices you use to look into planningalerts?

This will help us learn what devices we need to support in our project.

We know that the current system only works from wide screen devices/“desktop” because you just couldn’t flick between council websites and google sheets and actually accomplish the task on a phone or tablet. I’ve sat with Pip, our main volunteer, and she’s been doing it on a laptop. When @archoo made a big contribution it involved running a scraper and all-sorts, so we can safely assume they weren’t on a phone. (correct us if we're wrong @archoo 😉 )

But—we also want to create something usable by people who use PlanningAlerts more broadly. We can get the device stats from Google Analytics 😄 :

Over the last 7 days sessions are very evenly split between wide and small screen devices, and touch/point and click input (mouse/trackpad) modes:

screen shot 2017-06-02 at 1 02 34 pm

We think people making comments might be people who want to update councillor information so they can write to them. What devices do people successfully making comments use? It's very evenly split as well:

screen shot 2017-06-02 at 1 24 09 pm

I think this means we should be aiming to make something that people can use no-matter what device that bring to the party. That's best practice for a long time anyway. We also design things mobile first any way, mostly because its easier than going the other way, and makes us focus on keeping things simple.

do you use keyboard shortcuts? What other things do you use your computer? How often do you use computer?

I seems to me that these three questions are about trying to work out how proficient contributors are with computers. This is useful to know because it will guide the types of features we can add.

From my experience with them, and @archoo’s feedback below, I think our contributors fall into two groups around their proficiency with data collection tasks:

  1. People who are programmers and are comfortable working with data in formats like JSON and CSV. @archoo is the example of this group so far, but there are a number of people in the OpenAustralia Foundation community that might want to contribute. People in this group explore and develop tooling to help with data collection they do.
  2. People who aren't programmers, but are have a lot of experience using computers to create and edit documents, email etc. in their work. They have lots of experience with collecting and entering data, but they don't use keyboard shortcuts, and aren't familiar with tools that could help or with developing them. Outside of work they're more likely to use smart phones or tablets for web browsing etc. . Pip Brown is a good example of people in their group. There's lots of people potentially in this group among PlanningAlerts users.

That's a simplistic distinction, but useful I think. There's people who are in the middle, and lots of people who are less familiar with data entry too. Two of our working Design Principles are:

  • Strive for universal accessibility
  • Make the process of contribution is obvious and intuitive

So I think we want to make something that people who are less proficient with computers than those two groups ☝️ can use too.

If you would to contribute, does region makes difference? Do you prefer to input data for councillors in your region?

@archoo is in Queensland and contributed data for four councils in that state. Pip Brown started with the councillors that represent them, but then contributed to lots of other areas.

The people who've notified us about out-of-date councillor records via PlanningAlerts have only done it for councillors in their area. They've been commenters concerned that their local councillor options weren't correct; and, councillors and council staff asking us to update it so they receive records correctly.

It seems there's a distinction here between volunteers and people PlanningAlerts

Volunteers like Pip want to contribute beyond just the records for their local area. @archoo was interested in doing it a number of councils in the state they're from.

People using PlanningAlerts, who don't even know this is a project they can contribute to, have only taken interest in their own local area. That could be because the task of updating the records is part of another task for them, like sending or receiving comments correctly (we're had messages from commenters and councillors like this).

How much time do you spend contributing at once?

Pip Brown worked on collecting councillor data in blocks of 2 or so hours. I imaging it took @archoo a few hours to work on scrapers and contribute data for four councils.

Other times people have just done one council at a time. This can still take 20 minutes or so.

What's stopping you from contributing at the moment?

This is a really good and important question @hisayohorie . I think we'll need to ask our current contributors directly about this. It's also the type of question that it's hard for people to answer because they need to describe their own motivations. But, let's give it a go. It might be useful to ask them about other volunteering that they do, and to try and compare why prioritise contributing to some things over others.

@archoo

This comment has been minimized.

Show comment
Hide comment
@archoo

archoo Jun 3, 2017

Contributor

Correct.. Desktop with many monitors ;) - I had written a previous set of scripts that would deal with a couple of councils.. So I refreshed those and ran them for the ones I could.. but in the end it was far simpler to just transcribe directly from each website.. And I had a lazy Saturday night, so I got through quite a few.. Given these things don't change regularly and the site layout could very well change between each election, it seems like a waste of effort to try and fully automate something that happens so rarely..

Having said that, I think this is a useful piece to carve off into its own solution.. I did do some research on the standard model being used to store them.. Popolo supports far more than is currently being recorded for this project.. I did start to rough out a physical database model to store the councillor data independantly as a single source of truth, with versioning and history using the full Popolo model, but as usual I got sidetracked..

I think this is where the effort is best spent.. design a mechanism/interface to store and maintain as much detail as possible and then farm out versions of it to the various OAF projects that may require it. With a coordinated repository and appropriate governance, it may even be possible to approach the relevant state organisations (eq LGAQ in Qld) to provide the data in a compatible format after each election..

Good luck.. Life isn't going to let me do much in the next month or two, but I remain an interested onlooker and will answer quesions when I can..

Contributor

archoo commented Jun 3, 2017

Correct.. Desktop with many monitors ;) - I had written a previous set of scripts that would deal with a couple of councils.. So I refreshed those and ran them for the ones I could.. but in the end it was far simpler to just transcribe directly from each website.. And I had a lazy Saturday night, so I got through quite a few.. Given these things don't change regularly and the site layout could very well change between each election, it seems like a waste of effort to try and fully automate something that happens so rarely..

Having said that, I think this is a useful piece to carve off into its own solution.. I did do some research on the standard model being used to store them.. Popolo supports far more than is currently being recorded for this project.. I did start to rough out a physical database model to store the councillor data independantly as a single source of truth, with versioning and history using the full Popolo model, but as usual I got sidetracked..

I think this is where the effort is best spent.. design a mechanism/interface to store and maintain as much detail as possible and then farm out versions of it to the various OAF projects that may require it. With a coordinated repository and appropriate governance, it may even be possible to approach the relevant state organisations (eq LGAQ in Qld) to provide the data in a compatible format after each election..

Good luck.. Life isn't going to let me do much in the next month or two, but I remain an interested onlooker and will answer quesions when I can..

@equivalentideas

This comment has been minimized.

Show comment
Hide comment
@equivalentideas

equivalentideas Jun 6, 2017

Member

@archoo Thanks for this :) That's really helpful. It sounds the project is going to be similar to what you were imagining.

We'll be sure to ping you when need be and we'll loop you into some early testing when it's time!

Member

equivalentideas commented Jun 6, 2017

@archoo Thanks for this :) That's really helpful. It sounds the project is going to be similar to what you were imagining.

We'll be sure to ping you when need be and we'll loop you into some early testing when it's time!

@henare

This comment has been minimized.

Show comment
Hide comment
@henare

henare Jun 6, 2017

Member

Having said that, I think this is a useful piece to carve off into its own solution.. I did do some research on the standard model being used to store them.. Popolo supports far more than is currently being recorded for this project.. I did start to rough out a physical database model to store the councillor data independantly as a single source of truth, with versioning and history using the full Popolo model, but as usual I got sidetracked..

I think this is where the effort is best spent.. design a mechanism/interface to store and maintain as much detail as possible and then farm out versions of it to the various OAF projects that may require it. With a coordinated repository and appropriate governance, it may even be possible to approach the relevant state organisations (eq LGAQ in Qld) to provide the data in a compatible format after each election..

This is what PopIt and its successor PopIt NG set out to solve. During this project we'll evaluate if they'd be useful to us here.

Good luck.. Life isn't going to let me do much in the next month or two, but I remain an interested onlooker and will answer quesions when I can..

Member

henare commented Jun 6, 2017

Having said that, I think this is a useful piece to carve off into its own solution.. I did do some research on the standard model being used to store them.. Popolo supports far more than is currently being recorded for this project.. I did start to rough out a physical database model to store the councillor data independantly as a single source of truth, with versioning and history using the full Popolo model, but as usual I got sidetracked..

I think this is where the effort is best spent.. design a mechanism/interface to store and maintain as much detail as possible and then farm out versions of it to the various OAF projects that may require it. With a coordinated repository and appropriate governance, it may even be possible to approach the relevant state organisations (eq LGAQ in Qld) to provide the data in a compatible format after each election..

This is what PopIt and its successor PopIt NG set out to solve. During this project we'll evaluate if they'd be useful to us here.

Good luck.. Life isn't going to let me do much in the next month or two, but I remain an interested onlooker and will answer quesions when I can..

@equivalentideas

This comment has been minimized.

Show comment
Hide comment
@equivalentideas

equivalentideas Jun 9, 2017

Member

This is what PopIt and its successor PopIt NG set out to solve. During this project we'll evaluate if they'd be useful to us here.

Just noticed that Sinar also have a UI project for PopIT NG:

Generic Editing and Viewing UI

This user interface is meant for users maintaining the database, to enter and edit information following Popolo standard classes and fields. It is not meant to represent the data in specific manner, other than as per the specification definitions. eg. It does not assume that an organization is a legislative assembly, a committee or company. It will simply present the data as per the Popolo specifications.

Goal is that a group planning to start adding information for open data governance database, as information is available to them in a flexible manner.

Status

Front-end at usable stage of development. https://github.com/Sinar/sinar_popit_ui

Member

equivalentideas commented Jun 9, 2017

This is what PopIt and its successor PopIt NG set out to solve. During this project we'll evaluate if they'd be useful to us here.

Just noticed that Sinar also have a UI project for PopIT NG:

Generic Editing and Viewing UI

This user interface is meant for users maintaining the database, to enter and edit information following Popolo standard classes and fields. It is not meant to represent the data in specific manner, other than as per the specification definitions. eg. It does not assume that an organization is a legislative assembly, a committee or company. It will simply present the data as per the Popolo specifications.

Goal is that a group planning to start adding information for open data governance database, as information is available to them in a flexible manner.

Status

Front-end at usable stage of development. https://github.com/Sinar/sinar_popit_ui

@equivalentideas

This comment has been minimized.

Show comment
Hide comment
@equivalentideas

equivalentideas Jun 9, 2017

Member

From our problem statement above:

  • The current systems requires a lot of programming knowledge to contribute the data. People don’t even know they can update the data.
  • Very few people are able to make contribution, and so very few people do.

What does that actually look like?

We're creating a flow of how a better system that does what we need could work, and @hisayohorie has written up the current flow so we can compare. I've filled out the admin side at the end:

Steps for Contributor:

  • discover this project
  • get to the GitHub page
  • read the instructions for updating the data
  • find a GitHub issue about which updates are needed
  • go to the Google Spreadsheet, and find the right sheet and section for the council to update
  • individually research on council’s website
  • find out which councillors left the job
  • add end_date for the councillors who left
  • add new councillors
  • open shell on local machine
  • git clone down this repo
  • run bundle, then bundle exec rake to generate popolo
  • git add --patch and do all the y and n to add the changes for one council at a time if you've updated more than one
  • git commit
  • git push
  • make pull request in GitHub

Now the admin picks it up:

  • looks at Pull Request in GitHub
  • checks the data to see if it looks right
  • merge the Pull Request or request changes
  • once the Pull Request is merged...

On local machine

On WriteIt admin

  • Login
  • Refresh the data source for file for the state that includes the updated council

On WriteIt public site

  • check that the new councillors are options to write to

On PlanningAlerts:

  • Login to admin
  • Go to the admin page for the authority
  • Click 'Load Councillors'
  • Check that the results are what you expect
  • Edit the authority
  • Turn on the Ask Your Councillors feature
  • Find a public application page for the authority
  • Check that you can write to the new councillors as expected

On GitHub:

  • Note on the PR that the councillors are loaded and live

On Twitter:

  • Celebrate the person who collected the data and let people know the can now
    write to their councillors.
Member

equivalentideas commented Jun 9, 2017

From our problem statement above:

  • The current systems requires a lot of programming knowledge to contribute the data. People don’t even know they can update the data.
  • Very few people are able to make contribution, and so very few people do.

What does that actually look like?

We're creating a flow of how a better system that does what we need could work, and @hisayohorie has written up the current flow so we can compare. I've filled out the admin side at the end:

Steps for Contributor:

  • discover this project
  • get to the GitHub page
  • read the instructions for updating the data
  • find a GitHub issue about which updates are needed
  • go to the Google Spreadsheet, and find the right sheet and section for the council to update
  • individually research on council’s website
  • find out which councillors left the job
  • add end_date for the councillors who left
  • add new councillors
  • open shell on local machine
  • git clone down this repo
  • run bundle, then bundle exec rake to generate popolo
  • git add --patch and do all the y and n to add the changes for one council at a time if you've updated more than one
  • git commit
  • git push
  • make pull request in GitHub

Now the admin picks it up:

  • looks at Pull Request in GitHub
  • checks the data to see if it looks right
  • merge the Pull Request or request changes
  • once the Pull Request is merged...

On local machine

On WriteIt admin

  • Login
  • Refresh the data source for file for the state that includes the updated council

On WriteIt public site

  • check that the new councillors are options to write to

On PlanningAlerts:

  • Login to admin
  • Go to the admin page for the authority
  • Click 'Load Councillors'
  • Check that the results are what you expect
  • Edit the authority
  • Turn on the Ask Your Councillors feature
  • Find a public application page for the authority
  • Check that you can write to the new councillors as expected

On GitHub:

  • Note on the PR that the councillors are loaded and live

On Twitter:

  • Celebrate the person who collected the data and let people know the can now
    write to their councillors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment