New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dat Module Wishlist #5

Open
maxogden opened this Issue Jul 24, 2014 · 37 comments

Comments

Projects
None yet
@maxogden
Member

maxogden commented Jul 24, 2014

This issue is to discuss ideas for dat modules that would be awesome to have. If you want to write a module leave a comment to notify others that you are working on it (preferably also leave a link to the repo).

I'll keep this post updated to link to all of the modules (until we have too many to fit here).

To learn more about dat modules check out the Importing data guide

If a module doesn't have a link w/ it it means nobody is working on it yet

Read about Import Methods to determine which type of module should be used for each case.

List

(not in any particular order)

In Progress/Done

Up for grabs

@maxogden maxogden referenced this issue Jul 24, 2014

Closed

geopipes #140

@missinglink

This comment has been minimized.

Show comment
Hide comment
@webmaven

This comment has been minimized.

Show comment
Hide comment
@webmaven

webmaven Jul 24, 2014

Some other storage possibilities (in no particular order, and I am not working on any of these):

Some other storage possibilities (in no particular order, and I am not working on any of these):

@maxogden

This comment has been minimized.

Show comment
Hide comment
@maxogden

maxogden Aug 7, 2014

Member

@webmaven awesome list, thanks! I definitely have @jbenet on the hook to write an IPFS backend once it's ready

Member

maxogden commented Aug 7, 2014

@webmaven awesome list, thanks! I definitely have @jbenet on the hook to write an IPFS backend once it's ready

@webmaven

This comment has been minimized.

Show comment
Hide comment
@webmaven

webmaven Aug 7, 2014

You're very welcome, @maxogden.

webmaven commented Aug 7, 2014

You're very welcome, @maxogden.

@astro

This comment has been minimized.

Show comment
Hide comment
@astro

astro Aug 19, 2014

Here's one uploader for ElasticSearch based on the module by @missinglink: https://github.com/astro/dat-elasticsearch-upload

astro commented Aug 19, 2014

Here's one uploader for ElasticSearch based on the module by @missinglink: https://github.com/astro/dat-elasticsearch-upload

@thadguidry

This comment has been minimized.

Show comment
Hide comment
@thadguidry

thadguidry Aug 19, 2014

Support the CKAN API, in particular through http://datahub.io (part of CKAN - Open Knowledge Foundation) - Latest docs here: http://docs.ckan.org/en/latest/api/index.html

Support the CKAN API, in particular through http://datahub.io (part of CKAN - Open Knowledge Foundation) - Latest docs here: http://docs.ckan.org/en/latest/api/index.html

@calvinmetcalf

This comment has been minimized.

Show comment
Hide comment
@calvinmetcalf

calvinmetcalf Aug 20, 2014

pouch-dat is a connector for pouchdb/couchdb and dat

pouch-dat is a connector for pouchdb/couchdb and dat

@knownasilya

This comment has been minimized.

Show comment
Hide comment
@knownasilya

knownasilya Aug 20, 2014

rethinkdb might be a good one to add as well

rethinkdb might be a good one to add as well

@wardi

This comment has been minimized.

Show comment
Hide comment
@wardi

wardi Aug 20, 2014

+1 on CKAN datastore API support, or even a CKAN extension that uses Dat instead of CKAN's datastore.

wardi commented Aug 20, 2014

+1 on CKAN datastore API support, or even a CKAN extension that uses Dat instead of CKAN's datastore.

@wardi wardi referenced this issue Aug 20, 2014

Open

Dat integration #75

@RichardLitt

This comment has been minimized.

Show comment
Hide comment
@RichardLitt

RichardLitt Aug 21, 2014

Contributor

Neo4j might be another possibility.

+1 for CKAN.

Contributor

RichardLitt commented Aug 21, 2014

Neo4j might be another possibility.

+1 for CKAN.

@coleww

This comment has been minimized.

Show comment
Hide comment
@coleww

coleww Aug 25, 2014

https://github.com/coleww/s3-blobs I make no promises, but i shall try!

coleww commented Aug 25, 2014

https://github.com/coleww/s3-blobs I make no promises, but i shall try!

@doowb

This comment has been minimized.

Show comment
Hide comment
@doowb

doowb Aug 27, 2014

Not sure if this fits here, but I just created vinyl-dat to read/write dat records in a gulp workflow. It's basically just a way to convert dat records into vinyl files. I think it still needs some work and possibly a better way to just let dat do conversions on contents (like choosing csv, json, etc...)

doowb commented Aug 27, 2014

Not sure if this fits here, but I just created vinyl-dat to read/write dat records in a gulp workflow. It's basically just a way to convert dat records into vinyl files. I think it still needs some work and possibly a better way to just let dat do conversions on contents (like choosing csv, json, etc...)

@doowb

This comment has been minimized.

Show comment
Hide comment
@doowb

doowb Aug 27, 2014

firebase - I'll update if I start on this.

doowb commented Aug 27, 2014

firebase - I'll update if I start on this.

@jcblw

This comment has been minimized.

Show comment
Hide comment
@jcblw

jcblw Sep 7, 2014

I think this should work for redis as a backend https://www.npmjs.org/package/redisdown

jcblw commented Sep 7, 2014

I think this should work for redis as a backend https://www.npmjs.org/package/redisdown

@finnp

This comment has been minimized.

Show comment
Hide comment
@finnp

finnp Sep 8, 2014

I am going to have a look at CKAN as an import/export target as well as a possible storage back-end. ckanDOWN: https://github.com/finnp/ckanDOWN

finnp commented Sep 8, 2014

I am going to have a look at CKAN as an import/export target as well as a possible storage back-end. ckanDOWN: https://github.com/finnp/ckanDOWN

pvgenuchten added a commit to pvgenuchten/Cat-Interop that referenced this issue Sep 29, 2014

Update LinkPropertyLookupTable.csv
some project specific api's; google, osm
note that another potential collection of service-formats is here: datproject/discussions#5
@pvgenuchten

This comment has been minimized.

Show comment
Hide comment
@pvgenuchten

pvgenuchten Sep 29, 2014

Would https://www.npmjs.org/package/gdal be a usefull lib for dat to connect to a huge number of formats (http://gdal.org/ogr_formats.html)?

Would https://www.npmjs.org/package/gdal be a usefull lib for dat to connect to a huge number of formats (http://gdal.org/ogr_formats.html)?

@maxogden

This comment has been minimized.

Show comment
Hide comment
@maxogden

maxogden Oct 1, 2014

Member

@pvgenuchten that is awesome, thanks for linking it. I'm not sure what the best place to plug gdal into dat would be at the moment but I'll think about it. It's also great that the gdal module is using node-pre-gyp to make it easy to install cross-platform

Member

maxogden commented Oct 1, 2014

@pvgenuchten that is awesome, thanks for linking it. I'm not sure what the best place to plug gdal into dat would be at the moment but I'll think about it. It's also great that the gdal module is using node-pre-gyp to make it easy to install cross-platform

@markhahnel

This comment has been minimized.

Show comment
Hide comment
@markhahnel

markhahnel Oct 3, 2014

We'd be interested in throwing http://figshare.com/ in the ring here, also have some devs who could help out :)

We'd be interested in throwing http://figshare.com/ in the ring here, also have some devs who could help out :)

@calvinmetcalf

This comment has been minimized.

Show comment
Hide comment
@calvinmetcalf

calvinmetcalf Oct 3, 2014

So this is a pouchdb want originally but applies to dat.

Leveldown can be tricky to get working on all platforms and while pre-built ones ones are helpful there are always going to be weirder platform combinations.

It would be great to have a pure js leveldown backend (for node) that was not restricted to memory.

Being able to install dat or pouchdb with no c deps would be super useful.

Medea seems the best of what is out there but has a requirement that keys be kept in memory which combined with the node.js hard 2GB mem limit means that there are hard limits on the datasets you'd be able to put into it.

So really the thing that would be super helpful is a pure js nodejs db with spill to disk.

So this is a pouchdb want originally but applies to dat.

Leveldown can be tricky to get working on all platforms and while pre-built ones ones are helpful there are always going to be weirder platform combinations.

It would be great to have a pure js leveldown backend (for node) that was not restricted to memory.

Being able to install dat or pouchdb with no c deps would be super useful.

Medea seems the best of what is out there but has a requirement that keys be kept in memory which combined with the node.js hard 2GB mem limit means that there are hard limits on the datasets you'd be able to put into it.

So really the thing that would be super helpful is a pure js nodejs db with spill to disk.

@missinglink

This comment has been minimized.

Show comment
Hide comment
@missinglink

missinglink Oct 6, 2014

Check out http://geogig.org/

"GeoGig is an open source tool that draws inspiration from Git, but adapts its core concepts to handle distributed versioning of geospatial data."

Check out http://geogig.org/

"GeoGig is an open source tool that draws inspiration from Git, but adapts its core concepts to handle distributed versioning of geospatial data."

@kevinsimper

This comment has been minimized.

Show comment
Hide comment
@kevinsimper

kevinsimper Oct 12, 2014

Has many of the basic one not been covered? :) As seen here https://github.com/rvagg/node-levelup/wiki/Modules#storage

Or am I am wrong? I am a little confused because how is this dat related, isn't is more of Leveldb related, that is needs more adapters?

Or is there a good example on a plugin there is written?

Has many of the basic one not been covered? :) As seen here https://github.com/rvagg/node-levelup/wiki/Modules#storage

Or am I am wrong? I am a little confused because how is this dat related, isn't is more of Leveldb related, that is needs more adapters?

Or is there a good example on a plugin there is written?

@taterbase

This comment has been minimized.

Show comment
Hide comment
@taterbase

taterbase Oct 22, 2014

Just published dat-mongo to handle realtime updates from MongoDB with the listen hook.

Just published dat-mongo to handle realtime updates from MongoDB with the listen hook.

@yoshuawuyts

This comment has been minimized.

Show comment
Hide comment
@yoshuawuyts

yoshuawuyts Oct 23, 2014

@calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs

@calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs

@calvinmetcalf

This comment has been minimized.

Show comment
Hide comment
@calvinmetcalf

calvinmetcalf Oct 23, 2014

Ooo I like, thanks
On Oct 23, 2014 2:04 PM, "Yoshua Wuyts" notifications@github.com wrote:

@calvinmetcalf https://github.com/calvinmetcalf If you're looking to
build a pure JS levelDB this might be a good place to start --
https://github.com/rvagg/leveljs


Reply to this email directly or view it on GitHub
#5 (comment)
.

Ooo I like, thanks
On Oct 23, 2014 2:04 PM, "Yoshua Wuyts" notifications@github.com wrote:

@calvinmetcalf https://github.com/calvinmetcalf If you're looking to
build a pure JS levelDB this might be a good place to start --
https://github.com/rvagg/leveljs


Reply to this email directly or view it on GitHub
#5 (comment)
.

@wikier

This comment has been minimized.

Show comment
Hide comment
@wikier

wikier Dec 11, 2014

any interest to RDF / Linked Data support? I can work on a dat-marmotta implementation for testing it.

wikier commented Dec 11, 2014

any interest to RDF / Linked Data support? I can work on a dat-marmotta implementation for testing it.

@yueyingcui

This comment has been minimized.

Show comment
Hide comment
  • Couchbase
@almereyda

This comment has been minimized.

Show comment
Hide comment
@almereyda

almereyda Feb 1, 2015

@wikier RDF / Marmotta here, please! We'll need it for http://transformap.co

@wikier RDF / Marmotta here, please! We'll need it for http://transformap.co

@wikier

This comment has been minimized.

Show comment
Hide comment
@wikier

wikier Feb 2, 2015

so @almereyda, should we create an issue to organize the work, define the goals, etc?

wikier commented Feb 2, 2015

so @almereyda, should we create an issue to organize the work, define the goals, etc?

@almereyda

This comment has been minimized.

Show comment
Hide comment
@almereyda

almereyda Feb 6, 2015

@wikier Yes, of course! We'll be squatting the CouchDB Meetup in Berlin next week, where I'm also inviting @finnp right now, to precise the requirements of the TransforMap data integration pipeline.

Please feel free to get started with your perspective, but I will only then be able to respond meaningfully.

@wikier Yes, of course! We'll be squatting the CouchDB Meetup in Berlin next week, where I'm also inviting @finnp right now, to precise the requirements of the TransforMap data integration pipeline.

Please feel free to get started with your perspective, but I will only then be able to respond meaningfully.

@almereyda

This comment has been minimized.

Show comment
Hide comment
@almereyda

almereyda Feb 12, 2015

In the dawn of tomorrow, I'm stumbling over dat browser.

In the dawn of tomorrow, I'm stumbling over dat browser.

@almereyda almereyda referenced this issue Feb 12, 2015

Open

DAT Browser #40

@shawnbot

This comment has been minimized.

Show comment
Hide comment
@shawnbot

shawnbot Feb 20, 2015

I can't believe nobody's mentioned SQLite. So I'll do it.

I can't believe nobody's mentioned SQLite. So I'll do it.

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Jun 7, 2015

How about JSON, RDF, and Atom feed URLs? :-D

balupton commented Jun 7, 2015

How about JSON, RDF, and Atom feed URLs? :-D

@almereyda

This comment has been minimized.

Show comment
Hide comment
@almereyda

almereyda Jun 7, 2015

Yeah, then I could transform Discourse' API into RSS :) Or we just plug dat
on SocketHub. Or these feeds better live there.

On 7 June 2015 at 16:10, Benjamin Lupton notifications@github.com wrote:

How about JSON, RDF, and Atom feed URLs? :-D


Reply to this email directly or view it on GitHub
#5 (comment)
.

Yeah, then I could transform Discourse' API into RSS :) Or we just plug dat
on SocketHub. Or these feeds better live there.

On 7 June 2015 at 16:10, Benjamin Lupton notifications@github.com wrote:

How about JSON, RDF, and Atom feed URLs? :-D


Reply to this email directly or view it on GitHub
#5 (comment)
.

@fabianvf

This comment has been minimized.

Show comment
Hide comment
@fabianvf

fabianvf Aug 18, 2015

Guide links in the issue are broken.

Guide links in the issue are broken.

@onbjerg

This comment has been minimized.

Show comment
Hide comment
@onbjerg

onbjerg Nov 19, 2015

links to the guides are still broken

onbjerg commented Nov 19, 2015

links to the guides are still broken

@webmaven

This comment has been minimized.

Show comment
Hide comment
@webmaven

webmaven Nov 19, 2015

@maxogden, perhaps edit the original comment on this issue to use [ ] [x] checkboxes instead of bullets?

@maxogden, perhaps edit the original comment on this issue to use [ ] [x] checkboxes instead of bullets?

@patcon

This comment has been minimized.

Show comment
Hide comment
@patcon

patcon Feb 10, 2016

+1 on CKAN

patcon commented Feb 10, 2016

+1 on CKAN

This was referenced Jun 17, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment