New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipfs vs dat #119

Closed
scriptjs opened this Issue May 10, 2016 · 10 comments

Comments

Projects
None yet
9 participants
@scriptjs

scriptjs commented May 10, 2016

ipfs and dat seem to have a good deal in common. The purpose of dat seems to be scientific endeavour but it seems that ipfs could fill this void just as easily for small or large file collections and data sets. I am assuming ipfs will remain open source and free.

Is there a relationship between ipfs and dat? If so how might these work together since at this point they appear to be solving the same problem with a similar toolset.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet May 10, 2016

Member

Hey! Yeah actually. ipfs and dat are related projects, with many shared goals. IPFS is a lower level solution, while dat is more focused on the higher level data workflow pipeline questions. Aside from that, a lot of the tooling is similar because the problem space is similar. IPFS was originally more focused on versioning large files (typical machine learning data sets), and how now moved to just think about general merkle-linked data (data structures) with file importers. dat was originally more focused on tabular data (csv type), and now i think is moving towards versioned files? (im not super up to date).

As a fun fact, I actually worked on the dat project for some time in early 2014, to transfer a lot of what i wanted to do on data over to dat: http://juan.benet.ai/data/ . And both our teams are good friends! 👍 in general we share a lot of goals and we could probably serve the community better by working more closely. that was less amenable to dat in 2014 and early 2015, as IPFS was just getting off the ground, but perhaps now this is more possible. I for one would like to.

I personally think much of our efforts could be unified, based on efforts like IPLD, and our file importers, but i haven't had much success in convincing max. You can see various discussions throughout github or irc. I think we all have different things pulling at us and at the end of the day we all have to ship what we think is best for our users. But with enough modularity and useful standards we will all benefit. 😎 i think there's some work on us to land IPLD as a formal spec (more than just a readme spec) and make some good implementations in js that the dat team can use -- we're close to done on that. same goes for the file importers. less done on this.

Member

jbenet commented May 10, 2016

Hey! Yeah actually. ipfs and dat are related projects, with many shared goals. IPFS is a lower level solution, while dat is more focused on the higher level data workflow pipeline questions. Aside from that, a lot of the tooling is similar because the problem space is similar. IPFS was originally more focused on versioning large files (typical machine learning data sets), and how now moved to just think about general merkle-linked data (data structures) with file importers. dat was originally more focused on tabular data (csv type), and now i think is moving towards versioned files? (im not super up to date).

As a fun fact, I actually worked on the dat project for some time in early 2014, to transfer a lot of what i wanted to do on data over to dat: http://juan.benet.ai/data/ . And both our teams are good friends! 👍 in general we share a lot of goals and we could probably serve the community better by working more closely. that was less amenable to dat in 2014 and early 2015, as IPFS was just getting off the ground, but perhaps now this is more possible. I for one would like to.

I personally think much of our efforts could be unified, based on efforts like IPLD, and our file importers, but i haven't had much success in convincing max. You can see various discussions throughout github or irc. I think we all have different things pulling at us and at the end of the day we all have to ship what we think is best for our users. But with enough modularity and useful standards we will all benefit. 😎 i think there's some work on us to land IPLD as a formal spec (more than just a readme spec) and make some good implementations in js that the dat team can use -- we're close to done on that. same goes for the file importers. less done on this.

@scriptjs

This comment has been minimized.

Show comment
Hide comment
@scriptjs

scriptjs May 10, 2016

@jbenet I appreciate your reply. Yes, it seems it has moved to a more generic space to simplify the interaction between users and data with dat's latest code. I see great potential in both and would love to see these technologies layered with collaboration. It would really help to make this space clearer and more integrated as a whole. Synergy would also increase the momentum and visibility of both projects I believe.

scriptjs commented May 10, 2016

@jbenet I appreciate your reply. Yes, it seems it has moved to a more generic space to simplify the interaction between users and data with dat's latest code. I see great potential in both and would love to see these technologies layered with collaboration. It would really help to make this space clearer and more integrated as a whole. Synergy would also increase the momentum and visibility of both projects I believe.

@fedorov fedorov referenced this issue Jun 29, 2016

Closed

19 ipfs testing #22

@fedorov

This comment has been minimized.

Show comment
Hide comment
@fedorov

fedorov Jun 29, 2016

Anyone has experience comparing dat vs ipfs in terms of stability, not the features/underlying technology? This is what I am most interested in.

fedorov commented Jun 29, 2016

Anyone has experience comparing dat vs ipfs in terms of stability, not the features/underlying technology? This is what I am most interested in.

@fedorov

This comment has been minimized.

Show comment
Hide comment
@fedorov

fedorov Jun 29, 2016

image

there is no response trying to get my content from ipfs, and I've just started playing with it ... (sorry if this particular comment is out of scope for this FAQ entry - feel free to delete)

fedorov commented Jun 29, 2016

image

there is no response trying to get my content from ipfs, and I've just started playing with it ... (sorry if this particular comment is out of scope for this FAQ entry - feel free to delete)

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Jun 29, 2016

Member

@fedorov could you stop by IRC where we can help figure out whats not working?

Member

whyrusleeping commented Jun 29, 2016

@fedorov could you stop by IRC where we can help figure out whats not working?

@aaronshaf

This comment has been minimized.

Show comment
Hide comment
@aaronshaf

aaronshaf Jan 3, 2017

Can someone familiar with the recent dat explain the difference between ipfs and dat? @joehand? @maxogden?

Edit: https://github.com/datproject/docs/blob/master/docs/faq.md#how-is-dat-different-than-ipfs

IPFS and Dat share a number of underlying similarities but address different problems. Both deduplicate content-addressed pieces of data and have a mechanism for searching for peers who have a specific piece of data. Both have implementations which work in modern Web browsers, as well as command line tools.

The two systems also have a number of differences. Dat keeps a secure version log of changes to a dataset over time which allows Dat to act as a version control tool. The type of Merkle tree used by Dat lets peers compare which pieces of a specific version of a dataset they each have and efficiently exchange the deltas to complete a full sync. It is not possible to synchronize or version a dataset in this way in IPFS without implementing such functionality yourself, as IPFS provides a CDN and/or filesystem interface but not a synchronization mechanism. In short, IPFS provides distribution of objects, Dat provides synchronization of datasets.

In order for IPFS to provide guarantees about interoperability, IPFS applications must use only the IPFS network stack. In contrast, Dat is only an application protocol and is agnostic to which network protocols (transports and naming systems) are used. As a result, Dat cannot make the same types of interoperability guarantees as IPFS.

aaronshaf commented Jan 3, 2017

Can someone familiar with the recent dat explain the difference between ipfs and dat? @joehand? @maxogden?

Edit: https://github.com/datproject/docs/blob/master/docs/faq.md#how-is-dat-different-than-ipfs

IPFS and Dat share a number of underlying similarities but address different problems. Both deduplicate content-addressed pieces of data and have a mechanism for searching for peers who have a specific piece of data. Both have implementations which work in modern Web browsers, as well as command line tools.

The two systems also have a number of differences. Dat keeps a secure version log of changes to a dataset over time which allows Dat to act as a version control tool. The type of Merkle tree used by Dat lets peers compare which pieces of a specific version of a dataset they each have and efficiently exchange the deltas to complete a full sync. It is not possible to synchronize or version a dataset in this way in IPFS without implementing such functionality yourself, as IPFS provides a CDN and/or filesystem interface but not a synchronization mechanism. In short, IPFS provides distribution of objects, Dat provides synchronization of datasets.

In order for IPFS to provide guarantees about interoperability, IPFS applications must use only the IPFS network stack. In contrast, Dat is only an application protocol and is agnostic to which network protocols (transports and naming systems) are used. As a result, Dat cannot make the same types of interoperability guarantees as IPFS.

@kristojorg

This comment has been minimized.

Show comment
Hide comment
@kristojorg

kristojorg Apr 1, 2017

Can anybody tell me what is meant by interoperability here?

kristojorg commented Apr 1, 2017

Can anybody tell me what is meant by interoperability here?

@joehand

This comment has been minimized.

Show comment
Hide comment
@joehand

joehand Apr 2, 2017

Can anybody tell me what is meant by interoperability here?

The Dat protocol describes how two clients should interact once connected. However, that connection could be over a variety of networks. Dat doesn't make any guarantee, or have any requirement, that a specific network protocol is used for discovering other clients or exchanging data.

For example, Dat has a WebRTC implementation for use in browsers. Users on that network can only talk to each other, not clients running our command line application (which does not use WebRTC). We could make these two networks interoperable by creating a bridge between them but that is not part of the Dat protocol itself.

So you can create a Dat application using any type of network but that does not mean your application will work with other Dat application on different networks.

joehand commented Apr 2, 2017

Can anybody tell me what is meant by interoperability here?

The Dat protocol describes how two clients should interact once connected. However, that connection could be over a variety of networks. Dat doesn't make any guarantee, or have any requirement, that a specific network protocol is used for discovering other clients or exchanging data.

For example, Dat has a WebRTC implementation for use in browsers. Users on that network can only talk to each other, not clients running our command line application (which does not use WebRTC). We could make these two networks interoperable by creating a bridge between them but that is not part of the Dat protocol itself.

So you can create a Dat application using any type of network but that does not mean your application will work with other Dat application on different networks.

@kristojorg

This comment has been minimized.

Show comment
Hide comment
@kristojorg

kristojorg Apr 2, 2017

Ahhh I see thank you very much for taking the time to explain, that was extremely helpful!

kristojorg commented Apr 2, 2017

Ahhh I see thank you very much for taking the time to explain, that was extremely helpful!

@madavieb

This comment has been minimized.

Show comment
Hide comment
@madavieb

madavieb commented May 23, 2017

This issue has been moved to https://discuss.ipfs.io/t/how-does-ipfs-compare-with-x/465.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment