Skip to content

Config structure defs #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 36 commits into from
Feb 28, 2019
Merged

Config structure defs #8

merged 36 commits into from
Feb 28, 2019

Conversation

@andykais
Copy link
Owner

@andykais andykais commented Jan 26, 2019 •
edited
Loading

andykais added 30 commits January 24, 2019 15:32
@andykais
add handlebars unit test
fa1b07e
@andykais
config v2, basic build structure
30106d6
@andykais
convert to structure defs
ef0911a
@andykais
clean up lint
5866067
@andykais
remove unnecessary Rx.of observable
0bbc9f3
@andykais
hacky fix for ts-runtime/issues/21
c48e7bb
@andykais
tighter parser&downloader class types
c7685df
@andykais
add FMap wrapper around Map
c52a183
@andykais
Merge branch 'master' of github.com:andykais/scrape-pages into config…
50a4405 
…-structure-defs
@andykais
refactor runOptions into options
88ce72d
@andykais
adjust prettier width from 80 to 100
76392b0
@andykais
upgrade typescript-eslint
ac4c90d
@andykais
ensure input keys & scrapers are valid, unreserved names
8a924f9
@andykais
rename global testing vars
e55725f
@andykais
add pre-push check, fix handlebars vulnerability
dd71ea1
@andykais
serialize options & config
fff0cf2
@andykais
Merge branch 'config-structure-defs' of github.com:andykais/scrape-pa…
359d26b 
…ges into config-structure-defs
@andykais
simplify query output
ed5107e 
change query ReturnType from `{ [scraper:string]: Row }[]` to `Row[][]`
@andykais
add logfile rotation
81b02a2
@andykais
expose query.prepare to user
d4cf36c
@andykais
add osx to travis, move npm deploy inside .travis.yml
0b963a6
@andykais
add more fetch error catches, more map helpers
3d42562
@andykais
Merge branch 'config-structure-defs' of github.com:andykais/scrape-pa…
54b060c 
…ges into config-structure-defs
@andykais
add reusable options type
b5168f5
@andykais
add value limit per ScrapeConfig
ca16b4a
@andykais
push incomplete params third arg
f434200
@andykais
separate options & params
6a88f10 
options should be reusable, params is the truly one-off params
like input & folder
@andykais
add type assertion test
8347690
@andykais
remove lint
bc809ab
@andykais
rename config fields
ae1ee5a
@andykais
update readme
580a0e6
@andykais
add keyword to package.json
0495b2e
@andykais
remove import from config
f620897 
for now, this is not a pressing feature. See
#12 for details.
@andykais
rename url -> downloadData in sql
3cc622e 
replace mocha before function callbacks with async/await
@andykais
give query function to user before creating folders
b912e18 
big win to allow users to call `scrape` for both a scraper and a query
function #15
@andykais
update readme
168c2bc
@andykais andykais merged commit 6f61c4d into master Feb 28, 2019
@andykais andykais mentioned this pull request Feb 28, 2019
Closed
@andykais andykais deleted the config-structure-defs branch May 17, 2019 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

create query function before creating folders/tables validate input names & scraper names as slugs add limit flag to ScrapeConfig

2 participants

@andykais