Possible to add a dedicated url field ? #63

Open
jpfox opened this Issue Jan 14, 2014 · 38 comments

Projects

None yet
@jpfox
jpfox commented Jan 14, 2014

Hi,

Is it possible to have an url field outside of text content ?

140 chars is very short and using url shortner add dependency to an external service... It's a pity and in contradiction with a so decentralized architecture :-(

Thanks for your opinion about this idea.

@miguelfreitas
Owner

Yes, it is possible. But how do you think about the UI for that?
One possibility is that the UI would identify the url, remove it from the message and add it to the extra field.
However when one forwards the post to a legacy system the URL wouldn't be visible. Is this a good compromise?

@jpfox
jpfox commented Jan 14, 2014

I didn't think about multicast posting on several systems.
Do you think that adoption of Twister could be conditioned by content compatibility.
If you do, it's important to keep this compatibility.

Legacy system automatically replace long url with a short one, so long urls do not consume many characters in text. Keeping whole url in text for Twister is a huge limitation for user.
In the other side, how Twister should manage legacy content with legacy shortened urls ? Has it to keep this ugly short url inside text ?

More generally, a dedicated field would permit to escape from every legacy url shortners and maybe get target url when pasting a shortened one.

When forwarding to a legacy, tool (addon, UI or other) could concatenate text and url, and let legacy system shortening it its way. Friendica addon does it. Maybe Twister UI char counter could change color (orange) when it detects that text + legacy shortened url is too long to prevent user.

@BlockTester

I also think a short url mechanism is needed. It could handle both urls added to a post/dm or stand alone urls. URL shorter centralization is a huge problem.

@iShift
Contributor
iShift commented Jan 14, 2014

We can store it in DHT near twists

@BlockTester

Could something similar not be used to build in something like tweetmore?

@toyg
Contributor
toyg commented Jan 15, 2014

Considering Twitter actively forbids cross-posting with other networks, I'd say to hell with compatibility. The lack of proper metadata fields in twitter has always been one of its drawbacks, let's not repeat it here. A URL field would be perfectly fine, if it's not too technically difficult to implement.

@miguelfreitas
Owner

I'm not aware of this cross-posting prohibition.

Are you saying that if I want to duplicate my own post in two microbloggings they would forbid me of doing so? Sounds absurd. I can't give up on the copyright of my writting, neither they can claim exclusive copyright over my own posts.

@toyg
Contributor
toyg commented Jan 16, 2014

In order to post on Twitter via API you have to authenticate with OAuth, which is only possible for Twitter registered applications. One of the clauses in applications' standard TOS is that you will not use the app to synchronise with other social networks, end-of-story. This is a policy they implemented in the last 2 years as part of their big "crush 3rd party clients & other social networks" strategy -- they want people to use Twitter official clients and keep Twitter content inside Twitter. Of course you could register a "personal" app that would surreptitiously post your messages to Twitter and Twister, but if they found out, they would disable it (and potentially ban you). All big services that offered Twitter bridges (IFTTT etc) had to remove them.

@miguelfreitas
Owner

This is insane. They can't claim copyright over my own posts: if I copy-paste my post into this app using Twitter API and then also copy-paste it into my other twister app that would be legal, right?

@miguelfreitas
Owner

Btw, what about just reading posts from Twitter (not posting)? Do we need to sign a TOS as well?

@toyg
Contributor
toyg commented Jan 16, 2014

Yup, they disabled public RSS feeds, so you need an API key to do pretty much anything. And sure, you maintain copyright and can do copy-paste, what they won't allow is to do it programmatically via their API.
One could use a scraper, but you can imagine how painful that is.

This is why nobody does any Twitter integration anymore, except for the occasional post-update-to-Twitter from non-social applications.

@ezrafree
Contributor

Currently, I have an IFTTT rule that scrapes my Twitter feed and reposts any new posts to my Google Plus page (via BufferApp.com). But as toyg points out, I had to approve the app via their API first. To my mind, there shouldn't be a separate field for the URL (Twitter doesn't do that), but there should instead be some sort of URL-shortening taking place. Unfortunately, that would require DNS involvement to shorten a URL, so I can see why it hasn't been implemented. Still, it's something for us to be thinking about in my opinion. Perhaps there is some decentralized way of shortening links that only someone running twisterd can see the shortened link? Like a decentralized mini-dns system I'm imagining here. Kind of like what TOR does with their http://xxxxxxxxxx.onion/ URLs. So perhaps there could be http://xxxxxxxxxx.twister/ URLs?

@BlockTester

@ezrafree A simpler option not to have to deal and highjack the DNS might be to add a URL handler? shorttp://

@miguelfreitas
Owner

@ezrafree if we want a decentralized url shortener inside twister network we can easily add new DHT resources for it. For example:
./twisterd dhtput ezrafree shorturl_123 s "http://long-url-here"
The short link will need to be tied to your username though, perhaps using a different protocol like @BlockTester said: shorttp://ezrafree.123
The UI may convert it to the right link on the fly. Interesting ideas! :-)

@djmaze
Contributor
djmaze commented Apr 11, 2014

@toyg Did you see https://github.com/Astalaseven/twitter-rss? It is a scraper which works for my private needs, although it is not running perfectly stable.

@iShift
Contributor
iShift commented Apr 11, 2014

@miguelfreitas Good idea! it would be nice if you do this
my advice short url format: shttp://%SHORTURL%.%username%
Example%
shttp://AffA3c.shift/ = http://google.com/blablabla

@nitmir
nitmir commented Apr 11, 2014

If a url shortener is implemented in twister, it would be nice te be able to reduce any kind of urls (http://, ftp://, sftp://, irc://). It shouldn't be hard to had the scheme to the shortened url. For recall, the syntax of a url is scheme://domain:port/path?query_string#fragment_id

@thedod
thedod commented Apr 11, 2014

So far, I like the idea of @miguelfreitas best, but I'd suggest 3 modifications

  1. Since we're shortening, let's make the protocol short: sh://123.mfreitas (putting the user as "tld" is @[twister]rysiek's idea, and it makes sense).
  2. We can decide that the "default tld" is the author, so you could say sh://123 and save 9 letters ♠️.
  3. If we do ./twisterd dhtput ezrafree shorturl_123 s "some descrptive text|http://long-url-here" then the client has something to display as the link text or tooltip (although this might be a bit tricky for NoJS 😉).

@nitmir I think we should allow them all (BTW, about time someone started shorting magnet links), but it should be a well-known whitelist. We know we don't want javascript:// but what about cam:// (or ejectorseat:// 😜)?

@rysiekpl

OHAI, so I'm out of the woodwork.

  1. Anyway, I think I wouldn't go for the "default TLD, you don't have to use it if it's your link", as that would make some twists non-reshareable: suppose I make a 139-char twist with a sh://123; when somebody wants to re-share it, the link becomes sh://123.rysiek - and that gets the twist over 140 chars. So I'd say: use the TLD explicitly, always.
  2. As far as #NoJS clients are concerned, I guess it could be solved thus:
    1. get twist text
    2. if the twist contains an sh:// link, get the full link and replace it in the text
    3. display the twist with the replaced link.
  3. Do we want the 'sh' there? This might be non-transparent for new users, esp. those with some shell scripting experience. Maybe "link://" or "l://" (that's "small-L" there)?
@thedod
thedod commented Apr 12, 2014

So I'd say: use the TLD explicitly, always.

You've got me convinced.

if the twist contains an sh:// link, get the full link and replace it in the text

If we do it client side, it could take forever (this is why swizzler now fetches user details as iframes, and on the main page they're only mentioned as @usename). The way around this would be that the link opens /shorturl/thedod/123 in the search iframe. It shows long urt as a link, and — if @miguelfreitas implements it 👼 — the descriptive text that has been dhtputten (is that a word?) with the url.

Maybe "link://" or "l://" (that's "small-L" there)?

I think lowercase-L is an amiguity we can avoid by using u:// (u is short for "url-shortener-generated short url for use in string-length-constrained environments (e.g. Twister)" 😇)

@thedod
thedod commented Apr 14, 2014

@dryabov had an idea to couple the shortened links to a specific twist (dhtput them as an array). This got me thinking, why not a dict, and then it's zero markup? You write "my site has a new lolcat" and then a dict tells the client to convert "my site" and "lolcat" to links
local-scope-shortening

In fact, perhaps this can be done as part of the dhtput of the post itself. I'll go experiment a bit. I hope I won't break anything 😈

update: to do this - we'd need to rewrite createSignedUserpost() in twister-core. Not my department 😜

@rysiekpl

Oh this is a perfect way of dealing with it! No crazy url schemes, no extra characters like 'u://', etc. The only problem I see is when somebody posts a long link from a pure text client -- it should be changed into something short.

also, how would post editing look like? where do I write the dict data and should I use the {...} structure? this will be unusable for regular joes and jannettes, I think...

@thedod
thedod commented Apr 14, 2014

The only problem I see is when somebody posts a long link from a pure text client -- it should be changed into something short.

Why is that? The url never gets to be part of the twist. It's just an "overlay". In an SMS (for example) it would look like plaintext (maybe additional sms per link: lolcat: htt.... in case your phone can use that). BTW, SMS is no longer limited to 140 (even where I live 🐱 🐻 🐯).

also, how would post editing look like? where do I write the dict data and should I use the {...} structure? this will be unusable for regular joes and jannettes, I think...

Heavens forbid. editor would roughly look like

  Message: [I got a dog without a nose            ]
  links:
      [a dog  ][http://doge.orgy           ]
      [nose   ][magnet:.....               ]
      <add a new link>

Before fancy js editors were invented, such an interface was common for talk-backs in newspapers etc. (I don't think wordpress existed yet). I.e. end users (newspaper readers) understood how to use them

@thedod
thedod commented Apr 14, 2014

In the end it all boils down to [a refined version of] @jpfox's initial request. I propose an extended version of twisterd newpostmsg thedod 123 'the rain in https://en.wikipedia.org/wiki/Spain':

twisterd newlinkedmsg thedod 123 '{"msg":"the rain in Spain","links":{"rain":"...",...}}'

The userpost would look like unlinked text to an "old-skool" client (still has msg), and future clients would know to linkify it using links

{"userpost" : {
    "height" : 33103,
    "k" : 389,
    "lastk" : 388,
    "msg" : "The rain in Spain is case-insensitive",
    "links" : {
        "rain":"https://en.wikipedia.org/wiki/Rain",
        "spain":"https://en.wikipedia.org/wiki/Spain"},
    "n" : "thedod",
    "time" : 1397509203}}

Security heads up

We should have a whitelist of allowed protocols (not "anything but javascript:", because some devices have protocols for camera etc. so why guess what to blacklist?). We should also set limits on url size, number of links, etc. to avoid DoSing with a 1GB twist 😈

@miguelfreitas
Owner

@thedod currently i'm more interested in having an extended version of newpostmsg where raw "userpost" contents may be passed for signing and propagation. Or, better yet, receiving already signed content as mentioned in #4 to allow full web-clients solutions... that would be a huge win for widespread usage.

@iShift
Contributor
iShift commented Apr 16, 2014

@miguelfreitas mobile clients with twisterd is better for networks than client-server architecture
I hope that in the future we find IOS/Android developer that can write native client for that platforms (not web app) than push app to stores.

@thedod
thedod commented Apr 16, 2014

@miguelfreitas I don't see why we need to add JS encryption and weaken the system unnecessarily, and anyway - what would swizzler and curses clients do?

I think that twisterd should handle high level signing of json (as newpostmsg already does), and not count on clients to construct messages according to the protocol (they may be outdated, or buggy, or pwned).

I'd even consider removing signmessage from the rpc interface and only allow high level operations like "post" or "change profile", because the only way to handle unknown attacks is to whitelist what we define as acceptable.

For posting a twist, we already have newpostmsg that accepts a string payload, and I've proposed newlinkedmsg (including analysis of threats I could think of) as a possible alternative, but it could be anything else (e.g. call it newstyledmsg and also have fields like "bold":["call","have fields"]). The important thing is to keep creation of a signed message inside the twisterd box, where we can try to stop all known atacks (enforce size limits, avoid javascript: links, etc.) instead of counting on all clients to be up to date about it.

@miguelfreitas
Owner

@thedod you are thinking security the wrong way. adding json validation rules to twisterd when creating posts is useless. I own my twisterd client and i can do anything to it's code, including the removal of these validation rules. the only validation that really counts is the one you apply to the data that you receive from network, this is where stuff needs to be sanitized.

never trust that your attacker is running a good and well behaved twisterd copy that validated all json fields for you.

about having the JS encryption or not: i believe this is about whether we want to keep twister for a small group of hackers and users who install their own software, or if we want to reach the other 90% of users who will only access twister if it works inside their browser. of course we shouldn't trust on a site to deliver a good JS for you, so up to this point, the only solution i can think is installing a chrome app or similar. not as good as not installing anything, but at least we could be sure to be running a software that is authenticated... that's a different discussion though.

@miguelfreitas
Owner

@thedod if we had "change profile" as a written-to-stone twisterd RPC, now we wouldn't be seeing the tox and bitmessage fields that Calm theme added. i believe twisterd should help you, not prevent innovation and new ideas...

@thedod
thedod commented Apr 16, 2014

never trust that your attacker is running a good and well behaved twisterd copy that validated all json fields for you.

If there's a single "bad" client trying to post a post that doesn't comply with standards, I'd expect the "good" clients not to accept it. If this is not being done today, what's keeping an attacker today from posting 1GB twists and sinking the network down?

I certainly wouldn't want to reach a state where all known attacks should be handled by all known clients.

if we had "change profile" as a written-to-stone twisterd RPC, now we wouldn't be seeing the tox and bitmessage fields that Calm theme added. i believe twisterd should help you, not prevent innovation and new ideas...

Well, they got lucky 😉. As a counter example, you now have this request (where there's no way around it and it would require twisterd code changes). Wouldn't I like things to be easier for developers? Sure, but not at the expense of security. What would happen if profiles were "written in stone?", this would probably come as a pull-request, and would end up as a framework that makes it easy to pull-request the next one (top 10 recommended magnets?).

about having the JS encryption or not: i believe this is about whether we want to keep twister for a small group of hackers and users who install their own software, or if we want to reach the other 90% of users who will only access twister if it works inside their browser.

Why should end users care whether encryption was done in javascript or not? (also, what makes you think a NoJS web app can't be user friendly or that gateways to mail or im can't reach users in a "friendly" way, but even if "user friendly requires JS". Where does it say "JS encryption"?)

@miguelfreitas
Owner

If there's a single "bad" client trying to post a post that doesn't comply with standards, I'd expect the "good" clients not to accept it.

Exactly. That's what i've said (validate data coming from the network) not validating data before the post is signed as you first wrote.

If this is not being done today, what's keeping an attacker today from posting 1GB twists and sinking the network down?

Try it out ;-)

I certainly wouldn't want to reach a state where all known attacks should be handled by all known clients.

clients should be able to handle all network attacks. if they don't that's a bug!

Well, they got lucky 😉. As a counter example, you now have this request (where there's no way around it and it would require twisterd code changes). Wouldn't I like things to be easier for developers?

sure. but making it easier to developers that require more flexibility might be just to accept a raw "userpost" and signing it. i don't want to decide beforehand an RPC API for features we've not even dreamed about.

Sure, but not at the expense of security.

Exactly. Allowing clients to add arbitrary data to posts must never compromise security. If it does, then it is because we did something wrong.

What would happen if profiles were "written in stone?", this would probably come as a pull-request, and would end up as a framework that makes it easy to pull-request the next one (top 10 recommended magnets?).

A pull request, everybody has to compile their twisterds... check the other nodes you are connected to, there are still old clients around... Changing the client is much easier for everybody.

Why should end users care whether encryption was done in javascript or not? (also, what makes you think a NoJS web app can't be user friendly or that gateways to mail or im can't reach users in a "friendly" way, but even if "user friendly requires JS". Where does it say "JS encryption"?)

I never said the NoJS can't be user friendly.

But if you use a remote server from a third party without JS, then the encryption has to be performed at their server. In other words: you must trust the third party. That's not what i'm looking for, i'm look for end-to-end encryption.

@thedod
thedod commented Apr 17, 2014

If there's a single "bad" client trying to post a post that doesn't comply with standards, I'd expect the "good" clients not to accept it.

Exactly. That's what i've said (validate data coming from the network) not validating data before the post is signed as you first wrote.

Sorry. My mistake. I meant "bad/good" twisterd servers, but wrote "bad/good" clients. Let me say that again (I'll chance repeating myself, because IMHO this is important): as someone who's developing a client, if I can think of a possible attack, my duty is not to patch it for my client and advise others to do so (where? in a twist? the github wiki?). The right thing would be to open it as an issue at twister-core and make sure clients don't need to deal with this and introduce new errors in the process (maybe even willingly 😈).

It also doesn't mean these things should not be flexible, it just mean we need to sweat more in order to make them so.
We can have a flexible-yet-secure system with [for example] "semantic plugins". Such a plugin is simply JSON signed by a twister user (that's how the end user can decide whether to trust and install it).
The JSON would contain semantic definitions of fields (just a rough example off the top of my head):

{"plugin_type":"profile_field", "name":"blogroll", "fields":[
         {"name":"links", "type":"array",
             "opts":{"max":10, "type":"dict", "fields":[ ... ]}}, ...]}

Why do we need this? So that we can have [for example] the filed type "url". For twisterd it means [among other things] that the value should never begin with "javascript:", while for clients it should mean "I can put this as an href in a link and not worry about security because I got it from my twisterd".
If we need to trust all clients to plug all vulnerabilities once they get reported (where? at twister-core? at some client's repo? what if the maintainer's on holiday?), we'll end up like corporate software where zero-days never end because it's somebody else's department.

But if you use a remote server from a third party without JS

I'm not developing swizzler in order to turn it into a "federated system". Not sure what made you think that, but maybe my use of the word "server" was misleading:
Wherever I say server (as in server-side vs client-side), I mean "the twisterd running on localhost". As a client developer, this is what I see as "server", which brings me to:

A pull request, everybody has to compile their twisterds... check the other nodes you are connected to, there are still old clients around... Changing the client is much easier for everybody.

Only developers [during the alpha phase] need to compile. User will [eventually] have nice packaging with install/upgrade scripts and/or integration with existing software-delivery systems (apt, yum, appstores, etc.).
All this would require a lot of effort and contributors familiar with all kinds of distributions, and if twister gets popular enough, eventually twister-core will have automatic upgrades on many platforms, and "not too shabby" upgrade scripts on many others.

Now suppose a Calm user finds a problem, and hedgehog fixes it.How would calm users upgrade? (suppose they don't have fancy apt packaging, because their team is smaller than twister-core's). And then - how long before Swizzler finds out about it? Should all client developers be on a security alert mailing list?

@black-puppydog

So this just came up with @[twister]swisstengu and I am happy that there already is an issue for it.
But it seems to have gone wildly off-topic... Interesting talk, but not well-placed here IMHO.

So, what I take away from this is:

  • We want some kind of mechanism to shorten urls within the twister network
  • The idea so far is to either...
    • create a "spearate" entry for shortened urls in the DHT/Torrent networks and reference these
    • embed the shortened urls into the postings with some kind of markup
  • There might/should be some kind of control over which protocols are acceptable since we don't want anyone to abuse the links to brick our phone or such.

Personally, I would like to see this being solved like on twitter:
When I create a posting, all urls matching aforementioned format automatically get shortened to something... well, short. Maybe even just "[1]" or "[2]", but maye also more like on twitter, with a prefix of the urls of given size (I think twitter uses 22 chars or such?). Now, this can be done in twisterd, and if we don't use a NoJS client, then we can also do it in JS to give an accurate "remaining characters" value. But the JS client would then still send the text to twisterd varbatim, the urls would get shortened there, and attached to the DHT/Torrent posting.

As for any other regular posting, twisterd would make sure that posts it receives from peers match its specification of what a post should look like. I think so far this boils down to "valid signature, not too long" but then this specification would have to be extended by "not too many urls and none of them too long or with a known bad format"
But then again, I am not sure how much work this would require on the twisterd code.

In the frontend, this would make it possible even for a NoJS client to

  • load the whole posting
  • replace all shortened links with actual html links
  • use the expanded url as the tooltip of the html link or even just really display it (would look cluttered though, but hey, whatever the user wants...)
@thedod
thedod commented Oct 28, 2014

Thanks for summarizing this. Here are my remarks.

all urls matching aforementioned format automatically get shortened to something... well, short. Maybe even just "[1]" or "[2]"

I think there's a more intuitive (and backware-compatible) way: link [first occurrence of] a phrase to a URL.

Example: If twist is "Twister is a p2p social network. P2p rules!!!1" and the json also has

{"links": {
    "twister":"http://twister.net.co/",
    "p2p":"https://en.wikipedia.org/wiki/Peer-to-peer",
    "social network":"https://en.wikipedia.org/wiki/Social_networking_service"
}}

then you get a readable twist, and an easy way to linkify it (the 2nd occurence of "p2p" doesn't get linkified).

twisterd would make sure that posts it receives from peers match its specification of what a post should look like. ... "not too many urls and none of them too long or with a known bad format"

IMHO, It should be the client's responsibility to defuse malicious content.

It's not like twisterd shouldn't filter as well (to minimize damage to outdated clients, for example), but if we define the goal as "ensure clients can trust twist urls blindly", we can end up with malicious twisterd nodes, pre-zero-day twists (we can't delete by definition) and in general — a zero-day roller-coaster.

One more remark: "with a known bad format". IMHO it's better to whitelist (like "protocol can be http or https") than blacklist (like "protocol can't be javascript or any of those thingies that can cause known problems on known devices"), because black is the color of stuff we don't see very well 😉

@black-puppydog

I see how your proposed url coding scheme would make the postings more readable, but I don't see how they would be more backwards-compatible.
Do you mean backwards-compatible with a client that does not know about link shortening yet? It would completely hide the link. Or which kind of not-up-to-date client do you mean?

Personally I would prefer the twitter approach mainly because it is dead simple: Ctrl-V and done.
I regularly get annoyed by any kind of more sophisticated mechanisms, ranging from the "mildly annoying" markdown syntax to the "OMFG I WANT TO PUNCH THIS SOFTWARE IN THE FACE" of the WYSIWYG phpBB editors".
So personally (just to stress it once more, this is a matter of taste) I would aim for a no-brainer workflow at the cost of having url snippets in my postings. I actually look at the url prefixes popping up in my twitter TL before clicking anything, just to see if the link is even interesting for me.
But if we do it the right way, we might even be able to have both: a URL coding that allows me to just Ctrl-V my urls and get them aliased with a prefix, and you to markup your links to please your followers' eyes :)

For the security concerns:
I guess one can always link to an http:// site and then just redirect to cam:// from there, and it would be out-of-scope for twister to address this since it would be quite easy to circumvent possible countermeasures (see below for a first simple idea). If the phone/PC of the user just allows this kind of thing from a browser or such, then there is something wrong with the device/OS. In any case, even now the client should apply some decent filtering (whitelisting or blacklisting) but that is a separate discussion since it applies even if no url shortening is implemented at all. Just the specification of "known good format" would change if we implement ur shortening, image/video attachments or thelike. So maybe one should first see what is being filtered at the moment (I certainly don't know), where it is filtered and how, and then create a separate issue for that.

Example for how to mess with url whitelisting:

  • Post url to perfectly fine address: http://verfriendlyserver.com/ponyparty.html
  • Wait until followers have loaded posting. Since ponyparty.html is a harmless html page, no twisterd or client would block it, right? It is now hard to get out of the network again.
  • Replace ponyparty.html content with a redirect to cam://streamVideoTo?target=verfriendlyserver.com/ThankYouForYourCooperation
    Needless to say, if the client is clever, it will filter this. But really, this should be a concern of my Browser/Phone/Tablet to catch the last step.
    I am not a security expert, and this is just the most basic version of circumventing bad-link protections I could come up with.
@dryabov
Contributor
dryabov commented Oct 28, 2014

The first question we should answer is where to save full URLs. If we choose DHT, there will be no persistence, and data may be lost. If we choose blockchain, it will growth in size much faster than we have now, and sure there is no need to store ALL shortened URLs on every Twister node. And if we choose torrent (most reliable way IMHO), there will be no significant difference with current split post feature. Anyway, torrent's block size is 16Kb, so we have some "reserve" to store links alongside with message.

@thedod Just two cases related to your idea:

  1. Consider message "Get Twister at http://twister.net.co/" with {"links":{"http://twister.net.co/":"http://twi5ter.net/?ref=co"}} in metadata. Is it ok?
  2. Consider text from your example ""Twister is a p2p social network. p2p rules!!!1"and try to make second "p2p" instance to be a link and first one just a text.
@miguelfreitas
Owner

Excellent points by @dryabov. I will add some brainstorming here as well:

  • Yes, DHT is easier to store and use but it is harder to maintain persistence. Currently all posts are persisted in DHT for only a limited amount of time (~weeks/months), otherwise the network overhead in refreshing all posts ever produced would grow exponentially. So we have no choice other than prioritize some specific DHT content. imho knowing that shortened urls will eventually expire is not a satisfactory solution.
  • Blockchain is out of question.
  • User torrent is the best option imho. However we should try how fast we would be able to 1) use DHT to obtain the swarm members list from tracker, 2) successfully connect to at least one the swarm members and 3) request and receive the desired piece which contains the full url.

For users which we already follow, the torrent solution adds no delay at all. In fact, it would be actually faster than using DHT.

The problem i described above is the delay for non-followers. That's something we should try and see how it performs.

We may try a new protocol extension similar to http://www.bittorrent.org/beps/bep_0006.html "allowed fast" but where the requester would be able to choose the piece. Then how do we prevent abuse? I don't know, but running a cryptographic challenge might be an option. Sort of "do you want to resolve this url fast by obtaining such specific piece even if you are choked? then you must pay for it!"

Of course, storing to torrent doesn't mean we can't duplicate it on DHT, we can and we should do it. For recent posts, DHT would provide a faster retrieval mechanism for non-followers. When post gets older, the content retrieval would fallback to the slower torrent mechanism. We should implement a twisterd API that hides that complexity, something like this:

requestSpecificPost(username, postnumber) {
   if ( we are follower of username ) {
     return post_from_torrent(username, postnumber);
   } else {
     do 1 and 2 in parallel:
     1) start "dhtget username postnumber"
     2) start torrent(username), get post_from_torrent(username,postnumber)
     return whatever finishes first.
   }
}

remember that "post" may contain arbitrary content, either a message or a full url field etc. so this function would be useful for both linking specific posts and to the url shortener implementation.

@miguelfreitas
Owner

Basically the same arguments here https://groups.google.com/d/msg/twister-dev/oDKUr9oOBHg/6rzqqKoUCQAJ

(however this time i'm working on the actual implementation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment