Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
c3854bb
Start adding model for url logs
erinspace Jul 28, 2015
050e4ec
Remove the url log column - just adding to normalized
erinspace Jul 28, 2015
16c05c5
Add url processor to postgres processor
erinspace Jul 29, 2015
0921cc0
A few key error fixes
erinspace Jul 29, 2015
63243b7
Add invoke task to reset postgres and elasticsearch
erinspace Jul 29, 2015
66c1059
return things properly duh
erinspace Jul 29, 2015
3f88fee
Add inv alias to reset task
erinspace Jul 29, 2015
f132cd4
Merge branch 'develop' of github.com:fabianvf/scrapi into feature/url…
erinspace Jul 29, 2015
a16502f
Add actual uri to uri log
erinspace Jul 30, 2015
9d91fe3
Add post_processing flag, so we can more closely control post process…
erinspace Jul 31, 2015
1d9aae6
Update local-dist to have correct static files
erinspace Jul 31, 2015
581df5e
Move uri logging processor out of postgres processor
erinspace Aug 3, 2015
4bacb91
Change name of uri processor
erinspace Aug 3, 2015
48d3d89
Add basic processing tasks that do nothing right now
erinspace Aug 3, 2015
0c88127
update name
erinspace Aug 3, 2015
4350322
Add post log event
erinspace Aug 3, 2015
6436e77
Add inv task for process uris
erinspace Aug 4, 2015
558b853
Add celery task for process uri that does all documents
erinspace Aug 4, 2015
51be35f
Add try except for document normalized instead of the doc not existing
erinspace Aug 4, 2015
f264fb4
Remove unused os and django settings
erinspace Aug 4, 2015
5f50306
change args and kwargs for processor method
erinspace Aug 4, 2015
9db7b50
Add cross origin settings to local dist
erinspace Aug 4, 2015
8542dc0
Now specifiy source to just resolve urls in that source
erinspace Aug 4, 2015
ab02396
Massive if statement to organize source uris into common buckets
erinspace Aug 5, 2015
733c915
Merge branch 'develop' of github.com:fabianvf/scrapi into feature/url…
erinspace Aug 6, 2015
970cae2
Update logging to be by document instead of all at once
erinspace Aug 10, 2015
73d00bc
Start breaking into smaller chunks
erinspace Aug 10, 2015
da46999
Breaking URIs into smaller groups now by source instead of all one
erinspace Aug 10, 2015
6af8a54
Fix url parsing for all different types of URIs
erinspace Aug 11, 2015
2d8b6da
Change to uritype
erinspace Aug 11, 2015
8bac575
Update uri processing processor to be more universal
erinspace Aug 11, 2015
98b2be5
Celery task for processing uris updated
erinspace Aug 11, 2015
d6d96f9
Remove unecessary args from tasks
erinspace Aug 11, 2015
77b875c
Remove old comments
erinspace Aug 12, 2015
f380406
Add invoke apiserver task
erinspace Aug 12, 2015
1384544
Add proces one URI task that can be rate limited
erinspace Aug 12, 2015
3d851e1
Allow proper not async settings
erinspace Aug 12, 2015
89219ac
Update reset_all task to incl a confirmation
erinspace Aug 12, 2015
c44722c
Use Celery always eager setting for async control
erinspace Aug 12, 2015
86180f1
Pass around correct params to uri logging
erinspace Aug 12, 2015
6bfacb5
Merge branch 'feature/postgres_api' of github.com:fabianvf/scrapi int…
erinspace Aug 17, 2015
397af5d
Add basic scraping to uri pipeline
erinspace Aug 17, 2015
b1a3aec
Add basic scrapers for springer and science direct
erinspace Aug 17, 2015
83540d0
Move scrapers around, add more utilty
erinspace Aug 18, 2015
8c6f410
Leave things in an experimental state to get uri list
erinspace Aug 21, 2015
465a153
Merging changes
erinspace Aug 21, 2015
46f14da
Merge branch 'develop' of github.com:fabianvf/scrapi into feature/url…
erinspace Oct 6, 2015
4475e5d
Add models for a person and a URL harvest
erinspace Oct 6, 2015
386f85d
Make sure things are null that may be
erinspace Oct 6, 2015
12c0928
Remove generic id field and add some specific ones
erinspace Oct 6, 2015
20dcac6
Add back HarvesterResponse loss from bad merge
erinspace Oct 6, 2015
93292c0
Rename URI processing to helpers because it will happen in the proces…
erinspace Oct 6, 2015
690d089
Add the processing helpers to do some manipulating
erinspace Oct 6, 2015
73ac7f5
Add process_uris to postgres processor
erinspace Oct 6, 2015
d2c8820
Add back real processing, but leave testing writing to file just in case
erinspace Oct 6, 2015
926dc1f
Fix error in requirements
erinspace Oct 6, 2015
c1d7e38
Move scrapers for now
erinspace Oct 6, 2015
838b375
Add processing contributors to postgres
erinspace Oct 6, 2015
a4ca45b
Add task to process contributors
erinspace Oct 6, 2015
7712ff7
Add util function to gather contributors into one place for a source
erinspace Oct 6, 2015
eaacfd2
Add invoke task for process contributors
erinspace Oct 6, 2015
51e40e6
Update related name and add field for reconstructed name
erinspace Oct 7, 2015
7d2c060
Add contrib processig to events
erinspace Oct 7, 2015
6934afb
Add process_contributors to base processor
erinspace Oct 7, 2015
d0d1c84
Add process_contributors to base hold
erinspace Oct 7, 2015
2dabdbf
Add person gettr, and more to process contributors
erinspace Oct 7, 2015
eb87e3d
Fix process contribs nvoke task
erinspace Oct 7, 2015
f612961
Add registration app to settings
erinspace Oct 9, 2015
665cba5
Add registration urls to main proj
erinspace Oct 9, 2015
6e6ec50
Add all provider registration models, views, tests, and validators
erinspace Oct 9, 2015
d863f05
Add bower components
erinspace Oct 9, 2015
543ffd2
Custom css for registration page
erinspace Oct 9, 2015
987ba8a
Move registration tests file
erinspace Oct 9, 2015
815257a
Add registration urls to main urls
erinspace Oct 9, 2015
eabf5d7
Add pushed data models to api models
erinspace Oct 9, 2015
c617603
Add pusheddata serializers
erinspace Oct 9, 2015
edd77d0
Add pushedurls to granular webview app
erinspace Oct 9, 2015
c50b1fd
Add pusheddata views to webview views
erinspace Oct 9, 2015
457220e
Add bulk django to requirements
erinspace Oct 9, 2015
f3b3546
Add swagger and other static files for apiview
erinspace Oct 9, 2015
1be677c
Add api docs to templates
erinspace Oct 9, 2015
077a6b7
Add pusheddata permissions, schemas, and validators
erinspace Oct 9, 2015
71eb941
Push API tests
erinspace Oct 9, 2015
ad8829e
Remove long comments for urls
erinspace Oct 9, 2015
0e58dd6
Add restframework authtoken to installed apps
erinspace Oct 9, 2015
89fa69f
Add validation check for html instead of xml and empty docs
erinspace Oct 9, 2015
34b0ab3
Fix tests for registration app
erinspace Oct 9, 2015
339fb1b
New pg jsonfield actually does return dict, got rid of extra stuff
erinspace Oct 9, 2015
24b5f3b
Update tests
erinspace Oct 9, 2015
32c7f34
Add API vcr tests
erinspace Oct 9, 2015
7f3023c
Add migrations
erinspace Oct 9, 2015
0a62ab9
Indentation flake8 fix
erinspace Oct 9, 2015
e975350
Make document API views read only
erinspace Oct 12, 2015
f123231
Add desk email and password defaults to settings
erinspace Oct 19, 2015
be33fd5
Add base url for register
erinspace Oct 19, 2015
34ea653
Add compose desk email to utils
erinspace Oct 19, 2015
7e3533e
Add placeholder line for composing incomplete DESK email
erinspace Oct 19, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions api/api/settings/defaults.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@
'django.contrib.staticfiles',
'django.contrib.contenttypes',
'django.contrib.sites',
'rest_framework.authtoken',
'api.webview',
'api.provider_registration',
'rest_framework',
'corsheaders'
)
Expand Down Expand Up @@ -119,3 +121,7 @@
'osf.io',
'staging.osf.io'
)

DESK_EMAIL = 'setthis@cos.io'
DESK_PASSWORD = 'password'
FROM_EMAIL = 'setthis@cos.io'
9 changes: 8 additions & 1 deletion api/api/settings/local-dist.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,11 @@
}
}

STATIC_URL = '{}/static/'.format(DOMAIN)
STATIC_URL = '/static/'


CORS_ORIGIN_WHITELIST = (
'localhost:5000',
'osf.io',
'staging.osf.io'
)
48 changes: 48 additions & 0 deletions api/api/static/bower_components/bootstrap/.bower.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{
"name": "bootstrap",
"description": "The most popular front-end framework for developing responsive, mobile first projects on the web.",
"version": "3.3.4",
"keywords": [
"css",
"js",
"less",
"mobile-first",
"responsive",
"front-end",
"framework",
"web"
],
"homepage": "http://getbootstrap.com",
"main": [
"less/bootstrap.less",
"dist/css/bootstrap.css",
"dist/js/bootstrap.js",
"dist/fonts/glyphicons-halflings-regular.eot",
"dist/fonts/glyphicons-halflings-regular.svg",
"dist/fonts/glyphicons-halflings-regular.ttf",
"dist/fonts/glyphicons-halflings-regular.woff",
"dist/fonts/glyphicons-halflings-regular.woff2"
],
"ignore": [
"/.*",
"_config.yml",
"CNAME",
"composer.json",
"CONTRIBUTING.md",
"docs",
"js/tests",
"test-infra"
],
"dependencies": {
"jquery": ">= 1.9.1"
},
"_release": "3.3.4",
"_resolution": {
"type": "version",
"tag": "v3.3.4",
"commit": "a10eb60bc0b07b747fa0c4ebd8821eb7307bd07f"
},
"_source": "git://github.com/twbs/bootstrap.git",
"_target": "~3.3.4",
"_originalSource": "bootstrap"
}
Loading