Open Data Ship List

[andrew]

Todo list issue for #609 and https://github.com/librariesio/supporters/issues/9

• Decide on fields
• Export tasks
• Setup open data mailing list in mailchimp
• Documentation Open Data page #1485
• Create https://libraries.io/open-data as homepage for the release Open Data page #1485
• Data cleanup Open Data Ship List #1483 (comment)
• Add primary keys to projects, versions and repository csvs
• Test importing of all csvs to big query
• Collect quotes for promotion
• Make list of people to reach out to about launch
• Press release and announcement blog post
• Initial Export and upload to s3
• Get DOI numbers
• Decide on upload locations and mirrors
• Add to https://github.com/caesar0301/awesome-public-datasets

Related but not required:

• Improve API documentation
• Add list of API clients
• Add license information to footer of the site
• Add license information to API with commercial prompt to contact support@libraries.io

[andrew]

Tables to be exported:

• Projects
• Version
• Dependencies
• Repository
• Tags
• Manifests
• Repository Dependencies

Tables we won't be exporting as the data is available elsewhere in more reliable forms:

• Issues
• Readmes

We won't be exporting any User related data in this first release, namely these tables:

• Contribution
• Repository Users
• Repository Organisations

[BenJam]

The DOI number thing seems like it would be managed by Zenodo no? worth a check, otherwise CrossRef has the nicest logo of any doi provider 😬

[andrew]

First version of the export rake tasks are now merged: https://github.com/librariesio/libraries.io/blob/master/lib/tasks/open_data.rake

[andrew]

Some initial stats:

• projects
  • filesize: 649MB
  • compressed: 147MB
  • rows: 2,607,879
• versions
  • filesize: 932M
  • compressed: 136M
  • rows: 9,244,443
• dependencies
  • filesize: 228M
  • compressed: 147MB
  • rows: 49,826,914
• repositories
  • filesize: 5.7GB
  • compressed: 1.39GB
  • rows: 23,267,322
• tags
  • filesize: 5.9GB
  • compressed: 1.53GB
  • rows: 39,502,588
• repository dependencies
  • filesize: 8.75GB
  • compressed: 850MB
  • rows: 91,840,527

[andrew]

Currently investigating some invalid rows being generated in the projects csv file Fixed by removing newline characters from descriptions.

[andrew]

A folder containing all 6 csv files gzipped comes in at 4.3GB

[andrew]

Plans for data cleanup:

• check all projects to see if they've been yanked/removed (running now)
• check repositories with at least one dependency to see if they've been removed (running now)
• resync as many projects as possible (running now)
• resync most popular repositories

[andrew]

Thinking about adding back in the primary keys for projects, versions and repositories to make joining the data in the different csv files.

[BenJam]

I've created data@ and opendata@ btw.

[andrew]

Submitted pr to awesome-public-datasets: awesomedata/awesome-public-datasets#300

[andrew]

🚢