Database of the sources documents (i.e. ENB publications)

[rufuspollock]

Store at: data/documents.csv

What is wanted:

• id
• source url
• date
• event - event id from event table Database of Events #14

Do we already have this?

If not we can sort of get it from the txt files we have.

/cc @tommv

[tommv]

Yes we do have them:

• the name of the txt file is directly derived from the URL of the corresponding report (e.g. 1201000e.txt refers to http://www.iisd.ca/vol12/1201000e.html)
• This table (https://github.com/medialab/climateDebateExplorer/blob/master/ENB-data/metadata_overview/event_bulletinurl.csv) connect the URLs of the reports with the ID of the events
• and this other table (https://github.com/medialab/climateDebateExplorer/blob/master/ENB-data/metadata_overview/events_metadata.csv) connects the ID events with a bunch of meta-data on the events

[rufuspollock]

@pauloborges for this and the #14 (DB of events) we may just want to rescrape http://www.iisd.ca/enb/vol12/ as we get much more info e.g. issue no.

[pauloborges]

Just pushed this change. Check the script and both databases.

[rufuspollock]

@pauloborges looks good. Some quick comments:

• documents.csv
  • date column value is sometimes summary - that can't be right ...
  • I suggest document id is the file name with extension stripped - that way it corresponds to documents with have stored
• events.csv
  • suggest renaming name to title in column names
  • could we have title in the last column so stuff is more readable
• can we get a datapackage.json with those CSVs in it and description for columns. BTW https://github.com/okfn/dpm is your friend here ;-)

[pauloborges]

@rgrp,

Every event has one issue that is a summary of the event. It has no associated date, as you can see in the original page. What should I put as its value?

[pauloborges]

I've done the other suggested modifications, including adding a datapackage.json file. You can check it using the Data Package Viewer.

Please check the columns descriptions since my english isn't very good.

[rufuspollock]

@pauloborges i would put the date as the last day of the event.

[pauloborges]

@rgrp, ok! I'll change this as soon as possible.

[pauloborges]

@rgrp, done!

[rufuspollock]

FIXED.