Edgemesh P2P CDN crapware

[chrissnell]

URL(s) where the issue occurs

https://edgemesh.com/ and likely many others.

Describe the issue

Edgemesh is a Javascript product that lets website owners distribute the burden of their content distribution to the end users of this site. It does this by means of a Javascript client on end users' browsers. This crapware unwittingly turns site visitors into miniature CDN edge nodes and wastes users' bandwidth and disk space.

The script is currently typically edgemesh.client.min.js but some variations may exist.

More info can also be found on their Github project page: https://github.com/edgemesh/edgemesh.

Screenshot(s)

Invisible background Javascript.

Versions

• Browser/version: any
• uBlock Origin version: 1.13.8

Settings

• None

Notes

N/A

[smed79]

URL(s) where the issue occurs

https://marketsmedia.com/
https://classiccarclubmanhattan.com/

[gorhill]

The script is currently typically edgemesh.client.min.js but some variations may exist.

||edgeno.de^$script,third-party

and/or

/edgemesh.*.js$script

?

[chrissnell]

@gorhill Both together should do the trick. I think the second rule is much more likely to be used because I suspect most people will not use a third-party host for this.

[gorhill]

Question: why so many people thumbed up this issue? Is it being talked about somewhere?

[compuguy]

Yes, on hackernews. This issue is linked in a comment. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[chrissnell]

@gorhill https://news.ycombinator.com/item?id=15132811

[ekimekim]

Any chance this rule could not be put in the generic "uBlock filters" filter? I expect that list to only contain trackers, ads, anti-adblock, things that it's safe to say no-one wants. I don't expect it to block anyone trying to implement peer to peer content delivery. It should be under another list where I can make an informed choice about allowing this kind of behaviour, not buried where no-one will ever know, and can't easily allow it without also allowing a bunch of undesired, unrelated things.

EDIT: It's come to my attention their javascript may also do tracking. In which case, disregard the above.

[gotitbro]

Ok, this is sneaky. On their own website (https://edgemesh.com) they show a notice about using peering tech and that it might affect the bandwidth of the visitor but on the sites mentioned by @smed79 #659 (comment) no such notice is shown.

Really shoddy or sneaky work. Ideally such a thing should be opt in rather than opt out as is own their own website (no such option on the sites really using Edgemesh).

[gorhill]

It should be under another list where I can make an informed choice about allowing this kind of behaviour

If one care about informed choice, than edgemesh filters definitely fit in the default lists, even if there is no tracking involved. Just open the logger, select All, then visit the sites above pointed out by @smed79: tons of behind-the-scene requests, out of view of users. If one is fine with this behavior, than it's just a matter of pointing-and-clicking to create an allow (green) rule for edgeno.de in the popup panel. In effect, having these filters in uBO by default does bring back informed choice, formerly non-existent. There is no informed choice in having to opt-out of something you do not know is going on.

[SupremeTechnopriest]

Hello! I'm the Chief Technology Officer and Lead Engineer at edgemesh. What I think @chrissnell meant by "crapware" was "revolutionary emerging technology!" 😄

I wanted to note a few things and ask how it's best to handle this issue on our side:

Edgemesh does indeed use p2p (WebRTC) to distribute assets (images, videos, etc) but with the goal of DECREASING page load times not INCREASING them.

We don't "waste" bandwidth or disk space. Your extra bandwidth and storage capacity is temporarly utilized to provide everyone with a better experience (prioritizing you above all). By way of example, here is a screenshot of an example customer's page load times. Note that the edgemesh accelerated clients loaded the page ~40% faster than the non accellerated clients.

We are very committed to keeping our footprint extremely low on client devices and we take every measure necessary to ensure you never pay for someone elses bandwidth. As a matter of fact, when we do replicate assets we aim to make sure the payload is smaller than the original XHR request would have been. Below is a distribution of that over the past 90 days:

At the end of the day we are a client side, peer to peer distributed caching solution, so we do understand the concern. We always knew we would end up here (on your issue page) one day.

Edgmesh and uBlock share the same vision

First let me say that we very much believe edgemesh and uBlock share the same vision. We both want a clean internet, free from intrusion and censorship. Our goal was always to make the user experience better, and we’ve thus far delivered on that vision. The average page load time across our customers today is 37% faster with edgemesh than without and today we’ve helped customers offload well over 1 terabit of transit traffic in exchange of pre-cached mesh transfers. So in the cases where end user bandwidth is free, edgemesh helps customers lower their bandwidth fees – so they don’t have riddle their site with ads. I know of at least one instance where our CEO got back from a meeting with a media client and said he just gave them the software for free. When I asked why free he said “we read their content every day, wouldn’t it be nice if it loaded faster? They can either use the money to stop that painful popover ad or hire 10 more fact checkers. Either way, its better”.

The world is a big place - the internet frontier needs p2p

Offloading to save money is a common motivation in the US and Europe. In places like Sub Sahara Africa, p2p is the only real way to distribute content. Here the bandwidth capabilities intra-region dramatically outstrip the available bandwidth off country and its very expensive. A major reason for high (by US standards) per Mb pricing in SSA is due to the high cost (and 90%+ traffic rate) of off region (transit) traffic. Below is an example showing intra-region bandwidth vs. off-region bandwidth for different countries. Note the fastest growing (by internet penetration rate) are also the most unbalanced (Kenya, Rwanda etc).

Performance above all else, but you can always opt-out

You should never feel a performance hit (and if you do we really want to hear about it) as we are extremely careful not to block threads and we stay off the cpu for the most part. We have an fairly advanced method for detecting metered connections and edgemesh's replication functionality is only turned on after an unmetered connection is discovered. Okay, so all that said, you will always have people like @chrissnell who prefer not to share their resources. We totally understand that, which is why have added some better documentation around our opt-out mechanism. You can read about opting out here. We are of the belief that the site operator should be able to deliver assets to their users however they see fit. It's up to their team to decided if a user prompt is necessary. We provide a simple localStorage based mechanism which allows for prompted (by site owner) and unprompted (by user) opt-out. You can see an example of an opt-out prompt on our site.

We want to be transparent as possible with our customers and our customers users with out interrupting their onsite experience. We are working on a blog post that explains what you, as an edgemesh enabled site user, can expect to be happening inside your machine. Hopefully this will help to clarify some of the misconceptions on how our system functions. We work within the confines of the browser. Everything we are doing has been deemed OK by the engineers at Mozilla and Google. I think your gripe should be taken up with them. In reality a peer to peer opt in/out should be a browser level feature, and likely will be in the future.

Should we block all p2p?

I'm not sure that blocking edgemesh is the right course of action seeing as how an opt-out mechanism already exists. There are many other p2p solutions that exist in the wild that do not offer opt-out and are not as transparent as we are at edgemesh (we prefer not to name them so they dont end up on the blacklist too). Where do we draw the line? Should we ban all WebRTC transfers? Videos are ~20x the size of images, so just video?

Wrap up

We would like to formally appeal this issue.

If you deceide to keep edgemesh on your block list we will not attempt to skirt around your regex. In exchange, we request that you use a less strict regex. Currently you are blocking our application javascript (main.[hash].js) which prevents our site from rendering. The only file you will need to block is edgemesh.client.min.js. The name of this file will never change and it's always served from the same hostname sig.edgeno.de.

In response to @ekimekim's comment regarding tracking. We do record real user metrics unless a user has opted out. Blocking our tracking script seems more inline with uBlock's mission than blocking our client script. To block our real user metric beacon just add the file edgemesh.track.min.js to your block list and you will be golden. If you end up blocking the client script it will prevent the beacon from being injected. No need to block both.

We commend @chrissnell for sumbitting this issue and hope our opt-out mechanism helps to aleviate his concerns. We look forward to more suggestions and peer review. We encourage you to be active on our github issues page. We can address additional concerns and consider new feature propositions from there.

We also commend the uBlock team for their fast review and merge of this issue. As uBlock users ourselves, we would like to thank you all for your hard work and dilligence.

Thank you again for your consideration,

The edgemesh team

PS

If you are interested in learning more about how edgemesh works, check out our article "Cache me if you can" that was just published by the ACM. This article covers the edgemesh backplane. A new blog post will be out soon detailing the client side operations. Keep an eye on our blog for that.

[chrissnell]

@SupremeTechnopriest, honestly, you're still a long way from "acceptable" in my book.

My beefs with your technology:

Opt-out

I take issue with your opt-out philosophy. Even crappy old Octoshape back in the late Aughts required users to opt-in before sharing bandwidth. As I mentioned on HN, Octoshape was a total disaster for network administrators. Most average users do not run ad blockers and do not understand opt-out. Worse, your "opt-out" is barely even opt-out: the dialog is buried at the very bottom of your page where few users are likely to look! You offered up some excuses on HN for not implementing opt-in and one of them was an aesthetic objection to the opt-in mechanism. That's easy to fix: allow site admins to style the dialog.

Tracking by default

You're enabling these metrics to help with your marketing and operations efforts. You could potentially replace this centralized collection with your own set of spiders that crawl your customers' sites (and loading your JS) and measuring offload rate and speed, then extrapolate your global performance from these numbers. That's beside the point, however. Your software tracks by default and this taints your greater argument against block lists.

Unclear/unreliable bandwidth conservation measures

From what I've gathered on here and HN, you have identified some providers that meter bandwidth and you are using their AS numbers to exclude their users. You are also using the (new, experimental, only supported in Chrome > 61) browser Network Information API to detect metered/congestion conditions. Neither of these are foolproof and given the size of the Internet, the accuracy of your detection is almost certainly poor.

Effects on large corporate networks

You are forcing network admins at large sites to mitigate the effects of your software and that just doesn't sit well with me. Administrators don't have unlimited time and the onus should not be on them to have to build countermeasures to every new, sneaky P2P application.

Lack of consideration for your users' networks

You don't know with certainty what a user's network environment is like. The user might use a small ISP with a very strict no-P2P policy that you are not aware of. The user might be in an environment where there is heavy monitoring of network traffic and your software might raise someone's suspicion of them. You just don't know. You're opt-out by default so you can talk all you want about Internet improvement but this doesn't address the issue here.

What it would take for me to not want to block you

1. Make this software opt-in-only with deliberate consent (clicking a checkbox and clicking submit) and make it very difficult for site developers to override that. This would probably require renaming your JS source file and using a different domain because of the proliferation of the opt-out-only JS in the wild.

2. Remove all tracking code from your products and figure out a smarter, cleaner way to gauge network and product performance.

Nevertheless, this is @gorhill's project and it's him that you need to convince. I think your "make a better Internet" is pretty weak, personally.

[gotitbro]

I find it interesting how Edgemesh Corporation's employees have liked @SupremeTechnopriest post.

About the post, uBlock and Edgemesh are not the same. uBlock is all about user choice while Edgemesh with its opt-out structure is not, period.

[SupremeTechnopriest]

@chrissnell thanks for taking the time to write that outline. Based on what you have said here I don't think we are that far off from being acceptable! The two points you made are good ones. We are going to discuss opt-in as opposed to opt-out and go back to the drawing board on tracking.

@gotitbro only one of the likes on my post is from an edgemesh employee (@jacobloveless). The rest are likely hacker news readers following this issue.

@gorhill If you dont mind taking a look at the regex you have implemented to block our client we would appreciate it. Currently it blocks our application javascript (React) from loading. You can verify this by visiting edgemesh.com with uBlock enabled and opening the js console.

[SupremeTechnopriest]

@gorhill ah I just saw ba6ebcf. Thanks!

[x000ri]

@SupremeTechnopriest

There are many other p2p solutions that exist in the wild that do not offer opt-out

For example these one?

bem.tv
peer5.com
viblast.com
swarmify.com
streamroot.io
webrtc2cdn.io

[Hrxn]

I find it interesting how Edgemesh Corporation's employees have liked @SupremeTechnopriest post.

Fake news!

And by the way, I'm getting a bit tired of that dishonest reasoning when talking about the "morality" of Opt-out..

So far, the only really good point of criticism in here is the possibility of side effects on (large) corporate networks, and I remain a bit skeptical here if that is actually true.

[SupremeTechnopriest]

@x000ri to name a few yes. The ones you listed there are video only. We are one of a small few companies working with other content types.

[chrissnell]

Morality lectures from @Hrxn, an anonymous account with an anonymous picture and no noteworthy project contributions. LOL.

Clearly, this issue has struck a nerve. Edgemesh built a business on taking bandwidth from others without asking and now they've ended up--inevitably, I might add--on a major block list.

Boo hoo. Clearly, the guys at Edgemesh are quite intelligent. It's a shame that this is how they chose to make money.

[Hrxn]

Morality lectures from @Hrxn, an anonymous account with an anonymous picture and no noteworthy project contributions. LOL.

Ad hominem is all that you have to offer? Really nice, tough guy.

[chrissnell]

I've made my argument, in detail. Do you have something to refute it with, besides meaningless garbage attention-seeking exclamations like "Fake news!"?

Who are you and what are your credentials? Unlike you, I don't hide behind an anonymous and empty persona. You can google me and review my credentials. I've been working at ISPs and tech companies since the early 90s. I've run a lot of networks. That's my legitimacy. Where's yours?

[gorhill]

@chrissnell I am pretty sure he agrees with you. I think his comment referred to the thumbs up in @SupremeTechnopriest's comment. edit: well, re-reading, actually I have no clue -- anyways, below.

Just to be clear to all, thumbs up (fake or real) are completely irrelevant in my decision. I didn't yet answer because @SupremeTechnopriest's comment is huge and it's going to require a good chunk of contiguous free time for me to respond to it properly.

[SupremeTechnopriest]

Thanks for giving this your honest attention @gorhill <3

[SupremeTechnopriest]

Hey @gorhill! It's been a while. Just dropping by to let you know that Edgemesh is no longer a P2P solution. I think its safe to remove the /edgemesh.*.js$script,domain=~edgemesh.com|~edgeno.de rule from resource abuse.

We have moved away from p2p and now offer a managed service worker that is locked to a single domain. There is no more abuse of resources because we are only caching assets for the site you are visiting, making only the experiences you care about faster.

Please let me know if there are any objections and I can help to explain any parts of our tech that is unclear.

Thank you!