Cosmetic filters not always applied on specific website http://honyaku.yahoo.co.jp/

[runrunrirun]

Browser: Firefox 37.0.2
Other extensions include: NoScript, HttpsEverywhere

Having a problem pretty much since starting to use uBlock a couple of months ago with a specific site:
http://honyaku.yahoo.co.jp/

The problem sounds similar to one that was reported as addressed in 0.9.5.1 but is still happening for me with that release. What happens is that the cosmetic filters for this site I have defined simply are not applied after a restart of Firefox. Reloading the page also does not apply the filters. However, if I do something to my filter list to mark it as changed, Apply Changes, and then reload the page above, the filters are then applied and will continue to be applied until I quit the browser.

Here is my filter list. The element picker was used to create the cosmetic rules.

classic.wunderground.com###wu4TDU > a[href="//www.wunderground.com/weather-forecast/zmw:60193.1.99999"]
eow.alc.co.jp###AreaUpperLeft > .pr_text
honyaku.yahoo.co.jp###msthd
honyaku.yahoo.co.jp###sub-contents
honyaku.yahoo.co.jp###guide-bar
honyaku.yahoo.co.jp###g-navi > .cf
honyaku.yahoo.co.jp###judg
honyaku.yahoo.co.jp###translate-exp
honyaku.yahoo.co.jp###footer
honyaku.yahoo.co.jp###dic-sult > div > div > p
honyaku.yahoo.co.jp###dic-sult > div > p
||honyaku.yahoo.co.jp/darla/2-2-4_7/html/ext-render-secure.html
honyaku.yahoo.co.jp###judg > p
translate.google.com###gt-imld-promo > .gt-imld-promo-img
classic.wunderground.com###wu4TDU
translate.google.com###gt-community-promo
translate.google.com###gba
!jisho.org##body > .row:nth-of-type(1) > .columns.large-2 > .logo > a[href="/"]

[WyohKnott]

I've tried with honyaku.yahoo.co.jp###sub-contents and I have the same issue. I wonder if it's because #sub-contents is loaded dynamically. Disregard this, my own cosmetic filters elsewhere are not applied on reload too.

[WyohKnott]

@gorhill I can't see cosmetic-on.js or cosmetic-off.js injected anywhere?

[my-password-is-password]

Could this be it?

uBlock Origin 0.9.5.1

uBlock 0.9.3.1

Edit: uBlock Origin 0.9.5.1 not showing co.jp as 1st party anymore. The popup looks exactly like uBlock 0.9.3.1 now and the cosmetic filters are blocking. All I did was restart the browser. Going to see if I can reproduce the co.jp

[my-password-is-password]

Ok here are the steps to reproduce.

1. Open http://honyaku.yahoo.co.jp/ in one tab.
2. Observe the popup. Shows yahoo.co.jpin 1st party row
3. Open another tab to AMO and install an addon that needs to restart.
   • I used https://addons.mozilla.org/en-us/firefox/addon/restart-button/
4. Restart
5. Observe the popup for the honyaku.yahoo.co.jp tab. Now shows co.jpin 1st party row

The cosmetic filters don't apply after the restart. Not sure if this is what happened to @runrunrirun.

Here is a video. http://uux3.altervista.org/uBlockOriginRestart.webm

uBlock Origin 0.9.5.1
FF 37.0.2
Win XP

[gorhill]

the cosmetic filters for this site I have defined simply are not applied after a restart of Firefox. Reloading the page also does not apply the filters.

I can't reproduce. That will be the first thing for me do, find a case where I can consistently reproduce.

@runrunrirun When the problem occurs, can you tell me if you see the same quirk as @my-password-is-password sees in the dynamic filtering pane, i.e. see if the first party domain is reported as co.jp instead of yahoo.co.jp?

[my-password-is-password]

@gorhill I think so. When I click to noop co.jp the yahoo.co.jp below it gets nooped too. Is that how you wanted me to check?

This is what it looks like expanding the dynamic filtering pane:

[gorhill]

I want @runrunrirun to check if he get the same result as you. If the domain is reported as co.jp, that's definitely a problem.

[gorhill]

@my-password-is-password By the way I cannot reproduce what you do in your video.

[gorhill]

@WyohKnott These are injected manually, when clicking the cosmetic filtering switch in the popup.

[my-password-is-password]

@gorhill I can reproduce everytime. Maybe one of my other extensions. I really don't want to have to disable 30 addons. Dammit.

[gorhill]

@my-password-is-password What's your list of add-ons? I can try them one by one on my side.

[my-password-is-password]

@gorhill I restarted my computer and booted Win 7 and was able to reproduce with just uBlock installed. So maybe its a windows issue.

[gorhill]

For uBlock to see the domain of honyaku.yahoo.co.jp as co.jp means it was not able to load the Public Suffix List (located at assets/thirdparties/publicsuffix.org/list/effective_tld_names.dat).

I can't think of any reason why that would happen. Any error in the browser console about this?

[my-password-is-password]

@gorhill Just this in the console, ReferenceError: yFav is not defined. Don't really know how to use FF dev stuff. Trying to remember some about: url that Deathamns told me to use one time.

[gorhill]

Browser console is Ctrl-Shift-J.

[my-password-is-password]

Ok.

Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Firefox/chrome.manifest'.
Warning: attempting to write 4674 bytes to preference extensions.xpiState. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file.
Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Firefox/browser/extensions/%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D/chrome.manifest'.
"[CustomizableUI]" "Custom widget with id loop-button does not return a valid node" CustomizableUI.jsm:174
"[CustomizableUI]" "Custom widget with id loop-button does not return a valid node" CustomizableUI.jsm:174
Security Error: Content at moz-nullprincipal:{b4f0e476-bef2-4b61-a80d-383d4ccc2f14} may not load or link to chrome://ublock0/content/devtools.html?tabId=-1.
Use of getPreventDefault() is deprecated.  Use defaultPrevented instead. preload-min.js:1:0
ReferenceError: yFav is not defined honyaku.yahoo.co.jp:274:4
TypeError: n is null index.js:18:7846
A promise chain failed to handle a rejection. Did you forget to '.catch', or did you forget to 'return'?
See https://developer.mozilla.org/Mozilla/JavaScript_code_modules/Promise.jsm/Promise

Date: Wed Apr 29 2015 05:47:28 GMT-0700 (Pacific Standard Time)
Full Message: Error: loop is not enabled
Full Stack: this.MozLoopService.initialize<@resource:///modules/loop/MozLoopService.jsm:1078:29
TaskImpl_run@resource://gre/modules/Task.jsm:314:40
TaskImpl@resource://gre/modules/Task.jsm:275:3
createAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:249:14
LoopUI.init@chrome://browser/content/browser.js:4393:7
gBrowserInit._delayedStartup@chrome://browser/content/browser.js:11075:5
 MozLoopService.jsm:1078:0
1430311659034   Services.HealthReport.HealthReporter    WARN    No prefs data found.
JavaScript 1.7's let blocks are deprecated clipboard.js:129:6
JavaScript 1.7's let blocks are deprecated clipboard.js:140:6
Key event not available on some keyboard layouts: key="c" modifiers="accel,alt"

There was one line with a little badge to the right with the number 4 in it.

Security Error: Content at moz-nullprincipal:{b4f0e476-bef2-4b61-a80d-383d4ccc2f14} may not load or link to chrome://ublock0/content/devtools.html?tabId=-1.

Heres a screenshot:

[gorhill]

Alright, I can't reproduce in Windows (VM) either. So at this point I need someone with dev-fu who can reproduce to step into the code to find out why URI.domainFromHostname doesn't seem to work properly.

[gorhill]

Actually I've just got an idea of what might be happening... I will try to provide a dev version within one hour so this can be tested.

[WyohKnott]

This problem is relatively unrelated to OP's issues though?

The cosmetic filters are not applied. If I reload the page, there is no <style> with display:none in it despite the rule honyaku.yahoo.co.jp###sub-contents

[my-password-is-password]

@WyohKnott They do for me if I don't do the restart thing. In your request log you don't see any yellow rows if you fitler for 'dom'?

[gorhill]

This problem is relatively unrelated to OP's issues though?

It is related. The cosmetic filters involved are specific, i.e. thet apply strictly to honyaku.yahoo.co.jp. Internally, uBlock extracts the domain from honyaku.yahoo.co.jp to find out where the cosmetic filters are stored internally. These cosmetic filters are supposed to be stored in the yahoo.co.jp bucket. However, apparently uBlock thinks the domain of honyaku.yahoo.co.jp is co.jp. So it ends up looking up the wrong bucket. I believe this is caused by the little domain name caching mechanism in there which purpose is to avoid looking up the whole PSL to derive the domain name.

[WyohKnott]

@my-password-is-password I do not.

@gorhill I'm digging into retrieveDomainSelectors, called from contentscript-start for, I assume, getting the selectors to hide. The domain name is correctly resolved:

[gorhill]

@WyohKnott and what is being returned?

[my-password-is-password]

@gorhill I can reproduce in chrome too. Steps to reproduce.

1. Open tab to http://honyaku.yahoo.co.jp/
2. Do a chrome://restart/

It happened once. Can't do it any more uMatrix was getting in the way.

[WyohKnott]

@gorhill
As you can see, nothing. All the bucket = this.hostnameFilters[hash] conditions are false.
→ Because hostnameFilters doesn't contain any rule regarding honyaku.yahoo.co.jp despite being added to My filters. So the rule is not read correctly somewhere.

[gorhill]

@my-password-is-password Yes, I can reproduce this.

[gorhill]

New dev build released. This fixes the chrome://restart issue, so this fixes the domain name retrieval issue, and hopefully OP issue.

[gorhill]

@WyohKnott

The domain name is correctly resolved

Problem however is that the domain name at cosmetic-filters-dictionary-build-time might have been the wrong one.

[WyohKnott]

It doesn't for me:

! 29/4/2015 11:17:01 http://honyaku.yahoo.co.jp/
honyaku.yahoo.co.jp###sub-contents

is still not applied.

[gorhill]

It could be that the compiled version of custom filter list needs to be re-compiled, this time with the proper domain name. I should have forced a re-compilation of all filter lists. You can force re-compilation of custom filters by just changing something in it and click apply changes. Then afterward, see if the issue is back.

[WyohKnott]

@gorhill I did think of that. Erase all cosmetics, rebooted the extension, reapplied the filters. Still not working.

[gorhill]

@WyohKnott I don't know what to say then, as said, I can't repro, so I can't step in the code to find out what is wrong. For sure an issue was fixed here, as this fixed the case reported by @my-password-is-password , which case could lead to specific cosmetic filters being improperly stored.

If I could reproduce with your case, I would look at what the compiled filters look like for the user filter list, i.e. I would unselect all filter lists except the user one, and put a breakpoint where the compiled filter list is loaded, and look at its content (details.content).

[WyohKnott]

@gorhill I'll do a step-by-step of getCompiledFilterList when I can. When is it triggered?

[gorhill]

@WyohKnott Note that the fact that with OP forcing a reload of the filter list fixes the problem in his case suggest strongly that the issue was indeed a bad domain name being used at load time. So the fix here would fix his issue.

In your case you say forcing a reload of the filter list does not work? So that would be something else. Just to be sure, cosmetic filtering is not disabled for the site, neither globally?

[WyohKnott]

@gorhill No it's enabled everywhere.

[gorhill]

When is it triggered?

When the filter lists are reloaded, when the content of user filters changes, or when the user click Apply changes in the 3rd-party filters pane.

[gorhill]

@my-password-is-password Since you were able to reproduce in both cases (Firefox/Chromium), can you tell if the latest release helps?

[WyohKnott]

Details content contains some weird stuff:

"n�1�www�||ah�www.google-analytics.com/ga.js    xda-developers.comn�5�.�adf.lyn�41�widget�||ah�widget-cdn.rpxnow.com    foxnews.comn�1�google�||ah�google-analytics.com avianca.comn�41�tags�||_h�tags.tiqcdn.com/utag/*.js weather.comn�41�tags�||_h�tags.tiqcdn.com/utag/*.js cnet.comn�1�extras�||ah�extras.mnginteractive.com   twincities.comn�1�extras�||ah�extras.mnginteractive.com denverpost.comn�1�extras�||ah�extras.mnginteractive.com mercurynews.comn�5�.�statcounter.comn�31�.�cdn-static.liverail.comn�1�s0�||ah�s0.2mdn.net/instream/ wistv.comn�1�s0�||ah�s0.2mdn.net/instream/  nfl.comn�1�s0�||ah�s0.2mdn.net/instream/    cnet.comn�31�s0�||a�s0.2mdn.net/instream/flash/v3/adsapi_3.swfn�1�cdn�||ah�cdn-i.dmdentertainment.com   cracked.comn�1�cdn�||_�cdn.vurb.com/*/mixpanel.n�1�target�||ah�target.122.2o7.net   target.comn�1�adm�||ah�adm.fwmrm.net    go.comn�5�.�mopub.comn�5�.�safelinking.netn�5�.�outbrain.comn�5�.�inc.comn�1�intergi�||ah�intergi.com   tvgolo.comn�5�.�flurry.comn�31�s0�||ah�s0.2mdn.net/instream/    wsmv.comn�41�s0�||ah�s0.2mdn.net/instream/  wsmv.comn�1�google�||ah�google-analytics.com    jackbox.tvn�41�www�||ah�www.googletagservices.com/tag/js/gpt.js weather.comn�5�.�c.speedtest.netn�5�.�boldchat.comn�5�.�mixpanel.comn�1�google�1a�/google-analytics-dashboardc�h�2tl4�charlie.strim.io�#tweetsc�h�2tl4�ovh.strim.io�#tweetsn�1�cdn�||ah�cdn.turner.com/ads/nba/adfuel.js    www.nba.comn�1�google�||ah�google-analytics.com/analytics.js    www.vd.nln�1�.�yui.yahooapis.comn�5�.�top.mail.run�1�cdn�||ah�cdn-static.liverail.com   9to5mac.comn�15�css�||_�css.washingtonpost.com/*/ads.cssn�5�.�pcekspert.comn�5�.�clickbank.comn�1�awaps�||ah�awaps.yandex.ru    market.yandex.run�1�mc�||ah�mc.yandex.ru    market.yandex.run�1�yastatic�||_h�yastatic.net/market-export/*/advert.js    market.yandex.run�1�www�||ah�www.google-analytics.com/ga.js zie.nln�1�www�||ah�www.google-analytics.com/ga.js   onlinetv.run�5�.�carbonads.netn�1�imasdk�||ah�imasdk.googleapis.com twitch.tvn�1�2o7�||ah�2o7.net   totaljobs.comn�1�google�||ah�google-analytics.com   bikstok.sonymusicshop.dkn�1�google�||ah�google-analytics.com    musixmatch.comn�5�.�quantcast.comn�41�css5�||_h�css5.gaanacdn.com/minify-*/min/?    gaana.comn�5�keen�||_�keen.github.io/*/keen.min.jsn�1�cdn�||ah�cdn.phoenix.intergi.com  okgoals.comn�5�.�kissmetrics.comn�5�.�viewpoint.comn�1�i�||ah�i.cdn.turner.com/xslo/cvp/ads/freewheel/bundles/2/    cnn.comc�h�2tbl�ebay.co.uk�.pushdown-ad"

But nothing about yahoo.co.jp, and I don't have these sites in my cosmetic filters, nor I have visited them (i think). Edit: oh okay, it's assets/ublock/unbreak.txt, the only other thing I have activated.

[gorhill]

@WyohKnott These are uBlock's own filters. They are of no interest. As said, you need to unselect all filter lists except My filters, so this way you will know that the content of details.content is that of custom filters.

[my-password-is-password]

@gorhill I tested the latest uBlock Origin dev build on both FF and Chromium and its working good now. No more co.jp

[WyohKnott]

Okay it's official I'm dumb. /no problem here anymore.

[gorhill]

I won't ask, but I also made mistakes.

[gorhill]

So I am going to guess with good confidence this was the issue. This might explain the occasional failure reported by some people, there was definitely a sort of race condition at extension load time.

[WyohKnott]

Maybe OP had the same brain f*** as me, so we'll see when they come back.

[runrunrirun]

Wow, apologies for slowpoke, wasn't expecting such fast response. Just installed 0.9.6.0 and I believe I can confirm the issue seems to be mostly fixed. The only caveat is that the cosmetic filters are still not applied if the page is being restored at browser startup (i.e. saved tab being reloaded by Firefox). This behavior may be expected, since the filtered content was actually loaded and is in the cache and may be rendered before uBlock runs.

If I reload the page, filters are now applied every time without editing my local list. Personally, I can live with this.