Domains visited get leaked to DDG servers #527
Comments
|
Hi @Tritonio and thanks for your feedback. The purpose of the request you observed is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page. We use an internal favicon service because it can be complicated to locate a favicon for a website. They can be stored in a variety of locations and in a variety of formats. The service understands these edge cases and simplifies retrieval within our apps and our search engine. At DuckDuckGo, we do not collect or share personal information. That's our privacy policy in a nutshell. For more detailed information on that, you can checkout our privacy policy at https://DuckDuckGo.com/privacy. The favicon service, as with all our services, adheres to this privacy policy in that the requests are anonymous and do not collect or share any personal information. If you have further questions, please let me know. |
|
Hi @tagawa, |
|
I understand that there are different favicon standards and that in some cases it can be difficult to locate it.
|
|
Just to understand: Why can't the same algorithm, which is being used in the DDG endpoint, be utilized in the app? |
|
Hi there! If DDG reviews this, please use the header data on my web pages to determine what the favicon is. I do not have a default favicon for my site. My DeskThemePacks use PNGs for favicons based on the currently viewed theme's normal cursor. By looking for favicon.ico where it doesn't exist, it affects both my site's performance and SEO. |
|
If it was just search results, I wouldn't care. But this appears to be in use for rendering tabs that you're visiting, regardless of how you got there: (ETA: Or am I misunderstanding how this app is used? I'm not a smartphone user, and only use DDG in a regular browser.) |
|
It’s shocking to see how DDG staff responded to this issue. Doesn’t seem to care much about user privacy concerns. |
|
@CDRussell @tagawa This needs to be reopened and looked into. This is a serious privacy issue regardless of your "privacy policy" |
|
This made me really doubt privacy of DuckDuckGo. How it can be private and secure if you hire security experts who allow it to happen since 2019? "it can be displayed in certain places within the app or on the results page" |
[https://news.ycombinator.com/item?id=23708166] |
|
privacy should be the whole raison d'etre of DDG. time for a fork DuckDucGone ? |
|
I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual. |
@scabros - Please checkout the reply above: #527 (comment) This is not about using their search service, we all know search provider know exactly what you are searching for. But this particular issue is about using the browser, it's when user directly visit the website from browser address bar or reference link. |
|
So essentially, the trade-off is between performance and auditability of privacy. I predict that the user segment DDG appeals to will overwhelmingly prefer the latter. Yes, we already trust DDG, but only because we have to trust someone and others have proved to be untrustworthy. The issue isn't about whether the user trusts DDG, it's about minimizing the need for trust and maximizing the ability to verify privacy. Please consider reopening this issue. |
|
I think this would make this app qualify in F-Droid for the Tracking anti-feature, there is no good reason why every website URL should be sent to DDG servers. Regardless of intent, it does key bits of data, and there are better technical solutions to finding the favicon. |
|
I believe reopening and rectifying this design choice would go a long way to preserving the trust that DDG has worked hard to earn in the privacy-oriented community. Even if users trust that DDG is not using this data in any undesirable way, it would provide more peace of mind to users that the data not be sent in the first place. I am strongly in favor of changing this functionality to no longer leak visited domains. |
|
A long time DDG user and advocate here. First and foremost, thank you for building the most awesome search engine there is right now! 🙂 I must say that I am pretty appalled by this implementation. Yes, I trust DDG with my search data since I have to do searches and DDG always seemed like they actually care about privacy. But at the same time, a privacy aware browser should never send close to all the user activity to it's server, even if it is "only" the domains that are browsed. I think most DDG users could live with a slightly less then optimal display of favicons in order to gain the assurance, that their domain history is indeed private to their device. That latter is literally the main reason, why most people install your browser in the first place. I don't think this should be a discussion about "privacy policies". You are integrating a piece of code and infrastructure in your product, that has the capabilities to spy on people. It just shouldn't be there in the first place. You should not keep this issue just closed like this, this is a serious matter and it is obvious from the community reactions that this needs to be discussed. (https://news.ycombinator.com/item?id=23708166) |
|
How is this still closed? |
|
This "feature" has been added in March 2018: 03f99c4#diff-63ac5c0d645555fe179e72977d9c1728 And as of now it is still unchanged in the develop branch: (I did not look into the control flow to determine when this gets executed at all though.) |
There are plenty of good browsers on the market. This browser had once privacy as a main selling point, but as it looks, not any more. |
|
@tagawa : Trust is a very brittle thing. It grows slowly and takes years to built it. And only one line of code can shatter it. But instead to react professionally and contritely you made it worse to stamp on the shards to make sure no useful piece of trust will survive. You had one job! :-( |
|
Sorry but this is not enough reason. There is a simple question you should ask to yourself.
|
Meanwhile DDG literally collecting personal information from my device. 😕
|
|
For those that don't really understand what is happening: It is really about that code snippet from here: private const val faviconBaseUrlFormat = "https://icons.duckduckgo.com/ip3/%s.ico"
fun Uri?.faviconLocation(): Uri? {
val host = this?.host
if (host.isNullOrBlank()) return null
return Uri.parse(String.format(faviconBaseUrlFormat, host))
}The function for retrieving the Favicon is asking for the current host the user is on (e.g. apple.com) and sends it to a service hosted by duckduckgo.com that is saved in the const https://icons.duckduckgo.com/ip3/apple.com.ico I guess they have a cache for already visited hosts etc. This sends every host the user visits (or the Favicon is requested for) to the duckduckgo.com server. I think it does not really matter what you do or do not do with the data. Its not about trust – its about the minimum you need to trust and this is clearly breaking a lot of peoples perception of what the browser should achieve. As it currently stands, i would consider this even worse of what i imagine googles chrome would do to its users. To make it clear, i don't know exactly what google chrome is doing – its the perception of what i think certain vendors are doing, and i would not expect this from a "privacy first" browser from duckduckgo.com
Please help! |
|
99% of the websites I have seen have either
I really don't see the point of this URL anyhow. It's useful for search results (because you'd have to download every page to parse its favicon otherwise, and your search results are already served directly from DDG so there'd be no point), but using it in tabs is not the right way. I'm willing to say this is just laziness rather than malice, though: rather than writing code to figure out the favicon from the page, they just used whatever they normally used on the search results page. NOTE: DDG results page uses They also seem to be caching the most common favicons like #527 (comment) mentioned. i.e. visit "apple" on DDG and the top link has its favicon set as Please reopen and fix. |
|
What I find most disturbing is that fact that they haven't said a word on this topic for a year now. |
I would expect them to respond now that the link blew up on HN. |
first post says otherwise. |
|
We love DDG for privacy, not for favicons. Privacy starts from not collecting data. |
Not leaking every single domain a user visits is not a high bar. It's the bar you need to pass not to be considered spyware. FFS why are you people trying to defend an idiotic code decision? They are fixing it, great, I'm really happy for that. That doesn't mean we should trivialize the matter.
Leaking the domains to a server puts you leaps behind your competition and even your non competition since even regular browsers that don't focus on privacy end up being more private than you. DYOR but this is why I think these bugs are very serious: 40.114.178.124 is the IP that icons.duckduckgo.com resolves to and WHOIS says that it's managed by Microsoft. If this is Azure, which I would take a bet it is, then that means that for the last year or more, Microsoft could, technically tap into that server's memory to see who requested which domain and when. So not only do you need to trust DDG, you also need to trust Microsoft that they didn't do anything bad with the info. Does anyone still remember PRISM? This kind of passive access to user data is exactly what PRISM was about. Although I could be wrong and maybe the actual server is running in DDG's premises and Microsoft is simply acting as a reverse proxy. |
|
Seems like even if you could not fix the favicon serving issue, just deleting the favicon and not having any would be a better user experience. Given that putting the favicon up is breaking a core value proposition of your service. |
|
Hi all, CTO of DuckDuckGo here. I thought it might be helpful if I briefly shared some of our internal thinking around this issue so folks can see how we got here and how we plan to move forward. The logic behind how we’ve been displaying favicons in our apps is a function of how we operate our private search engine. Since we already have an anonymous favicon service through our search engine, using it has a number of benefits: it avoids more requests to known non-anonymous websites that are visited, it's way faster since it runs server side, saves user bandwidth, and the only externally visible attribute is that the app is connecting to DuckDuckGo.com (as the favicon location is actually encrypted in the path in transit). To our team, utilizing this anonymous service we had made for the search engine seemed like an optimal principled choice across a set of criteria. We want to be clear that at no point was the actual visited domain otherwise exposed. This favicon service is fully anonymous on our end, and URL parameters (like the favicon domain) are encrypted in transit, just like the search engine (with search queries). This is also why when this issue was raised in the past, we decided to keep the solution as is. At no point was this ignored. However, we understand that there is an alternative method of getting the favicons locally that a lot of folks prefer while still maintaining our privacy standards. We also believe that method is in line with our product vision of "Privacy, simplified.", considering its a somewhat simpler method than the one we had been using. So, we went ahead today and implemented the change for both Android (#878) and iOS (duckduckgo/iOS#667) that will move this logic onto the client, and we will no longer be using the favicon service in our apps. These changes are currently in the release phase and are rolling out live now. We really appreciate the feedback and exchange of ideas on this topic, and on ways to further improve the privacy of our products in general. |
That's quite a curious wording to talk about a solution that objectively raises privacy to a much higher standard. Please remember, my dear DDG: don't be Google. |
|
I'm personally surprised nobody has talked about the fact that DuckDuckGo's search engine has a strange redirection function when you click on a link. I would love to talk about that at some point, because I really don't understand why it's there. Many thanks, |
|
There's about seven red herrings in this huge thread. A lot of people who suddenly use DDG when it's convenient to complain about it. Right. @Tritonio was the only one whom I saw explain the actual vulnerability here (provided you have a positive bias of trust for DDG, which disclaimer, I do):
Ah, SIGINT. Spooky SIGINT. This quote points at Microsoft Azure's IP for example's sake, but that doesn't account for the countless ISPs your data travels through, even on a basic DNS level without vendor participation. A nefarious agency could MITM and use this as spyware for a correlation++ attack specifically targeting DDG users. That should terrify all of us because we love privacy, (right?) and when the right adversary arrives – "could" is usually a synonym for "will inevitably". I fully 100% believe it's innocent, I even feel it makes perfect sense in the mind of a rushed developer. I am happy to see it fixed. Now, with that said, I'd like to propose a solution so the product marketing people can be happy too. If you want to have this feature locally, make it secure and do the same thing – just bundle a massive SVG sprite sheet with the client (realistically, how big would it be?) and do some sort of table association to pull them up on demand if the hostname has a match. Otherwise, don't make an outbound network request, just return and wait! If you wanted to change the way you do it (and follow more-or-less standard practice), you can rely on site-provided webContent Anyway, I'll pass the mic off now. I think @nilnilnil and DDG exec team did a great job of fixing the issue. Up and onwards! |
|
@resynth1943 It's to not expose the referrer so that the site can't know what the user searched. |
This can be prevented by: |
|
Interesting. Thank you, @pcr910303. Personally, I'd love to see Referer die a death, it's about damn time. |
That is curious. When I click on a link in the search engine (here's one about ghosts), I get something like this: https://duckduckgo.com/l/?kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FGhost_(Swedish_band) <html><head><meta name='referrer' content='origin'></head><body><script language='JavaScript'>window.parent.location.replace("https://en.wikipedia.org/wiki/Ghost_(Swedish_band)");</script><noscript><META http-equiv='refresh' content="0;URL=https://en.wikipedia.org/wiki/Ghost_(Swedish_band)"></noscript></body></html>Couldn't this be used to track browsing history, too? Technically, query parameters are sent to the server. Wouldn't mind knowing more about this, if they would explain more about it. An interesting thought would be, if the user has JavaScript enabled, why couldn't DuckDuckGo cook up a URL that looks a little more like this: https://duckduckgo.com/l/#/kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FGhost_(Swedish_band) See that tiny change? That little hash prevents the URL from being sent to the server! I wonder how that would work with referers, too. Of course, parsing that hash would need a little bit of JavaScript, but if the user has it enabled, then... Maybe there's a technical reason why they use this redirection service, instead of EDIT: Looks like I was right. EDIT: Of course, this wouldn't be a problem if they used POST instead of GET, but I'm told Chrome doesn't play nicely with that. |
|
@resynth1943 You're right, the redirect is for backwards-compatibility reasons (there's more about it on this help page) but it can be disabled in the search settings, in which case just the Re. POST and GET, that can also be toggled in the same settings page with GET as the default. |
|
DuckDuckGo just went viral but in a very WRONG way. |
|
Come on guys, stop trolling on this post. DuckDuckGo already made an statement on this. Don't like it, fork it. Stop crying and demanding work from others. |
|
I am absolutely AMAZED on how people OVERREACTED about this. First disclaimer: I have NO connection to DDG, I am just a user. I work as sysadmin and web developer full time since 2004. I do agree this is a "feature" that could be implemented locally to let the users feel more comfortable with his privacy concerns. But this, for me, doesn't represent an anti-privacy practice, as a lot of people are complaining. DDG is basically a search engine. In my own phone I have DDG AND Firefox focus, and use the last as my default browser. Either way, again, I think is a good choice to make the requested change to the local implementation. What really irritates me is the tone some people come here, complaining in a way that is practically accusing DDG being the same as google & Co. Folks, the distance is HUGE, and i think behind all this there is a key point: it's not about if DDG is receiving your data or not (because, hey, they are a search engine, you know, if you don't input anything, the can't search anything for you). The key point here is what they DO with your data. And they ALWAYS were very clear about that. They anonymize it and process it, to have some valuable info that they could use to improve their products. The "favicon incident" from the users perspective, is just an example of another anonymized service for DDG. And then you have people coming here saying things like this:
All this people, use android? or they have hacked their phones and installed librem? could they all do a better job competing against google? To ALL DDG staff, thank you for being there, thank you for giving us an alternative to G, thank you for listening, thank you for being open and transparent and thank you for being condescendent even with those that dont deserve it (because you know there are many that do). |
Who's trolling? |
|
@scabros that's fair enough, dude, and I agree with that. People were rude, malicious and forming egregious remarks about the company that just made a mistake. I can see people have already down voted your post. I've got 31 down votes on a rational opinion, so good luck communicating with the children here. DuckDuckGo really isn't the second Google. Far from it. And that's why I use it. |
|
Unfortunately this isn’t the only “mistake” DDG has made: DuckDuckGo's privacy abuses-- current, historic, and by proxy. People may be overreacting to this post if this post represented a single mistake, but given the context of the post I linked, I suspect most haven’t reacted nearly strongly enough. Edit: link to hacker news instead of directly to the article |
|
First of all - I am a big fan of DDG
This is why I will now explain, (again) what the problem with your answer is and what it indicates.
Your 'internal thinking' is what has raised this issue in public! It is basically the bare reason for our concerns! Because it shows, that you do not even understand our privacy argument! It is not about the bug, it is your action and the, like @calimeroteknik puts it, "curios wording", that makes it appear, like you have not understood our critic at all. Time of Events
Bugs can happen! Bad decisions can happen! But arguing with "trust us, it anonymous", it "saves user badwidth", and its "the only visible attribute", on top of that "we already have a anonymous [...] service"! It is pure poison for trust, because it indicates, you entirely missed the point! Either on purpose or not. Both would be alarming!
Its not that we do not understand your reasoning or the reasoning of @tagawa. The problem is your reasoning, seems to entirely miss our point! Now you continue - as CTO! This indicates no real insight at all!
We said, please do not mix the search engine code and your browser code! They should not relate on each other if not necessary (for example if the user actually wants to do a search on DDG).
Like @calimeroteknik already said: Luckily higher security standards meet your standard! -_-
You only took this action after enormous public pressure, there was no sign of insight or internal intention to resolve this! This is why you have been compared with BigCorp, because they act like this. (Or like pointed out by some people, are even doing a better job than you, handling these kinds of issues) [LARP MODE ON]
You are welcome! I hope this helps! |
I overlooked the implications of this. It's much more interesting than it seems. Consequence 1:It means that DDG has started a game of
…with the general public. That's immoral, but I believe it's legal and therefore legally allowed to do this. (after all, DDG users are not paying customers) Consequence 2:The current evidence unambiguously proves that DDG is commiting at least one of:
A real plight, if you want to get away with only PR! I think that, in order of preference for the public, your options are: (1) This is a pretty effective psychological strategy, because it induces self-disgust in people who are not trained to fight it internally. Studying WW2 propaganda helps with understanding that one. My sharp words should not hide my encouragements towards duckduckgo fighting its innate corporate nature. |
FWIW Google has a service used in their Chromium browser (and also used by Firefox and others) for clients to find out whether a given host is malicious (e.g. a known phishing page). Their API can be used in a way in which hostnames are not sent to Google, and even Google Chrome does things that way. (Then again, there might be other features in Chromium sending hostnames to Google, such as the omnibox autocompletion) |
Everyone who keeps posting here, DDG already received a PR and merged at the codebase #878 Is non-sense to keep posting "furious" and "frustrated" comments here. Again, if people dislike DDG Android app in any sense, don't use it, or just fork it. The whole idea of OpenSource isn't treating developers as slaves who fulfill all demands of everyone, or to make threats at reddit or any other websites (posted bit above) full of conspiracy stuff and making claims of things they aren't true. This kind of behaviors of developers being kind of slaves from users must STOP. |
|
@waltercool the discussion here is not about the software. Admitting it was a mistake would be a start as far as this discussion goes, and that hasn't happened as far as I can tell. |
I do understand your concern, but please look the "issue" and "official response". Everything else happened after the ticket was closed, basically someone said: "Hey look this code you are leaking all url from users!", and they said "ehhm, no". A good way to report "a software issue" (because the original claim IS a "code" claim), would be creating a PR proposing a replacement of the DDG favicon server. Companies do opensource not only to expose their code to others, but for community to contribute bugs with change proposals. Reporting a bug doesn't solve anything, proposing solutions does. And following this, a claim related to "privacy concerns" and not "a code issue" happened at issue #877, which the reported posted a change request at Asana platform, and they accepted it. |
|
favicons can even be weaponized : https://github.com/devanshbatham/FavFreak |
|
DDG is no difference to google...no privacy at all #my opinion though |
|
I love how GitHub seems to be removing any and all functions that users can use to report abuse. Especially when it's a whole page of spam about video games (?) that's irrelevant to the project. Nevertheless, I hope DuckDuckGo can learn from the more fruitful responses, one of which has been authored by @calimeroteknik. I think DuckDuckGo's official response was quite... flattering to themselves, and their project. I'll seek to be more wary of products such as this in the future. |
|
Mates... you gotta do something. At least Brave admitted their mistake and pledged to never inject ref codes on crypto websites. And you, you send our history to DDG servers. And you dare to close the issue? |
and others
Android/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt
Line 60 in ed91c9e
This seems to be leaking all(?) the domains that users visit to your servers.
The text was updated successfully, but these errors were encountered: