Skip to content

Include data in the package #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 12, 2016
Merged

Include data in the package #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
df806dd
move files into module
eshellman Mar 9, 2016
87ce5d9
remove duplicates
eshellman Mar 11, 2016
eb985fa
need one more version
eshellman Mar 12, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 0 additions & 34 deletions MANIFEST
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion gitenberg/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@

__title__ = 'gitberg'
__appname__ = 'gitberg'
__version__ = '0.2.0'
__version__ = '0.2.3'
__copyright__ = 'Copyright 2012-2016 Seth Woodworth and the Free Ebook Foundation'

15 changes: 0 additions & 15 deletions assets/GITenberg_repo_list.tsv → gitenberg/data/GITenberg_repo_list.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6028,11 +6028,6 @@
6197 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-03_6197
6198 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-04_6198
6199 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-05_6199
6199 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-05_6199
6199 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-05_6199
6199 Cumner-s-Son-and-Other-South-Sea-Folk-Volume-05_6199
6200 The-Greek-View-of-Life_6200
6200 The-Greek-View-of-Life_6200
6200 The-Greek-View-of-Life_6200
6201 Cumner-s-Son-and-Other-South-Sea-Folk---Complete_6201
6202 When-Valmond-Came-to-Pontiac-The-Story-of-a-Lost-Napoleon-Volume-1_6202
Expand Down Expand Up @@ -6553,7 +6548,6 @@
6747 The-Log-of-the-Empire-State_6747
6748 Fulco-de-Minstreel--Een-historisch-verhaal-uit-den-tijd-van-Graaf-Jan-I-voor-jongelieden_6748
6749 Voyages-of-Samuel-De-Champlain-Volume-02_6749
6749 Voyages-of-Samuel-De-Champlain-Volume-02_6749
6750 The-Hawaiian-Archipelago_6750
6751 The-Winds-of-the-World_6751
6752 Study-and-Stimulants-Or-the-Use-of-Intoxicants-and-Narcotics-in-Relation-to-Intellectual-Life-__6752
Expand Down Expand Up @@ -7066,7 +7060,6 @@
7266 ----_7266
7267 Cardello_7267
7268 Monsieur-Bergeret-a-Paris-Histoire-Contemporaine_7268
7268 Monsieur-Bergeret-a-Paris-Histoire-Contemporaine_7268
7269 Macbeth_7269
7270 ----volume-4-10_7270
7271 ----volume-11-14_7271
Expand Down Expand Up @@ -7430,7 +7423,6 @@
7631 The-Disowned-Volume-01_7631
7632 The-Disowned-Volume-02_7632
7633 The-Disowned-Volume-03_7633
7633 The-Disowned-Volume-03_7633
7634 The-Disowned-Volume-04_7634
7635 The-Disowned-Volume-05_7635
7636 The-Disowned-Volume-06_7636
Expand All @@ -7443,7 +7435,6 @@
7643 Ernest-Maltravers-Volume-04_7643
7644 Ernest-Maltravers-Volume-05_7644
7645 Ernest-Maltravers-Volume-06_7645
7645 Ernest-Maltravers-Volume-06_7645
7646 Ernest-Maltravers-Volume-07_7646
7647 Ernest-Maltravers-Volume-08_7647
7648 Ernest-Maltravers-Volume-09_7648
Expand All @@ -7457,15 +7448,10 @@
7656 Kenelm-Chillingly-Volume-07_7656
7657 Kenelm-Chillingly-Volume-08_7657
7657 Kenelm-Chillingly-Volume-08_7657
7657 Kenelm-Chillingly-Volume-08_7657
7658 Kenelm-Chillingly---Complete_7658
7659 What-Will-He-Do-with-It-Volume-01_7659
7659 What-Will-He-Do-with-It-Volume-01_7659
7660 What-Will-He-Do-with-It-Volume-02_7660
7660 What-Will-He-Do-with-It-Volume-02_7660
7661 What-Will-He-Do-with-It-Volume-03_7661
7661 What-Will-He-Do-with-It-Volume-03_7661
7662 What-Will-He-Do-with-It-Volume-04_7662
7662 What-Will-He-Do-with-It-Volume-04_7662
7663 What-Will-He-Do-with-It-Volume-05_7663
7664 What-Will-He-Do-with-It-Volume-06_7664
Expand All @@ -7474,7 +7460,6 @@
7667 What-Will-He-Do-with-It-Volume-09_7667
7668 What-Will-He-Do-with-It-Volume-10_7668
7669 What-Will-He-Do-with-It-Volume-11_7669
7669 What-Will-He-Do-with-It-Volume-11_7669
7670 What-Will-He-Do-with-It-Volume-12_7670
7671 What-Will-He-Do-with-It----Complete_7671
7672 Harold-the-Last-of-the-Saxon-Kings-Volume-01_7672
Expand Down
0 assets/gutenberg_descriptions.json → gitenberg/data/gutenberg_descriptions.json
View file
Open in desktop
File renamed without changes.
7 changes: 6 additions & 1 deletion gitenberg/util/catalog.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import sh
import json
import os
import csv
from gitenberg.metadata.pg_rdf import pg_rdf_to_json
from gitenberg.metadata.pandata import Pandata
from gitenberg import pg_wikipedia
Expand All @@ -15,12 +16,16 @@
# sourced from http://www.gutenberg.org/MIRRORS.ALL
MIRRORS = {'default': 'ftp://gutenberg.pglaf.org/mirrors/gutenberg/'}

with open(os.path.join(os.path.dirname(__file__), '../../assets/gutenberg_descriptions.json')) as descfile:
with open(os.path.join(os.path.dirname(__file__), '../data/gutenberg_descriptions.json')) as descfile:
DESCS= json.load(descfile)

descs = {}
for desc in DESCS:
descs[desc['identifier'][32:]]=desc['description']
repo_list = []
with open(os.path.join(os.path.dirname(__file__), '../data/GITenberg_repo_list.tsv')) as repofile:
for row in csv.reader(repofile, delimiter='\t', quotechar='"'):
repo_list.append(row)

class CdContext():
""" A context manager using `os` to cd to a directory and back
Expand Down
2 changes: 1 addition & 1 deletion notebooks/delete repos.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1221,7 +1221,7 @@
" removed_ids.add(pg_id)\n",
"with open('../assets/GITenberg_repo_list_out.tsv','wb') as out:\n",
" writer=csv.writer(out,delimiter='\\t', quotechar='\"')\n",
" with open('../assets/GITenberg_repo_list.tsv','r') as f:\n",
" with open('../gitenberg/data/GITenberg_repo_list.tsv','r') as f:\n",
" for (pg_id, reponame) in csv.reader(f,delimiter='\\t', quotechar='\"'):\n",
" if pg_id not in removed_ids:\n",
" writer.writerow([pg_id, reponame])"
Expand Down
2 changes: 1 addition & 1 deletion notebooks/find_missing_repos.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@
" def __init__(self,_id):\n",
" self.book_id = _id\n",
"pg_ids=set()\n",
"with open('../assets/GITenberg_repo_list.tsv','r') as f:\n",
"with open('../gitenberg/data/GITenberg_repo_list.tsv','r') as f:\n",
" for vals in csv.reader(f,delimiter='\\t', quotechar='\"'):\n",
" pg_ids.add(int(vals[0]))\n",
"missing=[]\n",
Expand Down
4 changes: 2 additions & 2 deletions notebooks/remove_duplicates.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,7 +617,7 @@
"\n",
"pg_ids={}\n",
"dups=set()\n",
"with open('/Users/eric/github/local/giten_site/assets/GITenberg_repo_list_2016.02.12.tsv','r') as f:\n",
"with open('../gitenberg/data/GITenberg_repo_list.tsv','r') as f:\n",
" for vals in csv.reader(f,delimiter='\\t', quotechar='\"'):\n",
" if int(vals[0]) in pg_ids:\n",
" prev=pg_ids[int(vals[0])]\n",
Expand Down Expand Up @@ -645,7 +645,7 @@
" pg_ids[key] = newest.name\n",
" \n",
" \n",
"with open('/Users/eric/github/local/giten_site/assets/GITenberg_repo_list.tsv','r+w') as out:\n",
"with open('../assets/GITenberg_repo_list_out.tsv','r+w') as out:\n",
" csv.writer(out,delimiter='\\t', quotechar='\"').writerows([(key,pg_ids[key]) for key in pg_ids.keys()])"
]
},
Expand Down
1 change: 0 additions & 1 deletion requirements.pip
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,3 @@ isodate==0.5.1
pyparsing==2.0.3
rdflib==4.2.0
rdflib-jsonld==0.3
requests==2.7.0
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
'rdflib==4.2.0',
'rdflib-jsonld==0.3',
],

test_suite='nose.collector',
tests_require=['nose'],
classifiers=[
Expand Down