Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fanficfare -o use_browser_cache=true -o use_browser_cache_only=true with Flatpaked Firefox? #985

Closed
mcepl opened this issue Jul 31, 2023 · 28 comments

Comments

@mcepl
Copy link
Contributor

mcepl commented Jul 31, 2023

When I try to run FanFicFare from the main branch (specifically fb552c8) to get a story out of the Firefox cache I get this error:

firefox-test:firefox $ fanficfare -o use_browser_cache=true -o use_browser_cache_only=true -
o browser_cache_path=/tmp/mozilla-profile/cache2/ --debug 'https://www.fanfiction.net/s/1221
3201/1/Beautiful-Mind'
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(230):     OS Version:Linux-6.4.12-1-default-x86_64-with-glibc2.38
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(231): Python Version:3.11.4 (main, Jun 28 2023, 19:51:46) [GCC]
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(232):    FFF Version:4.27.0
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1056): use_nsapa_proxy:true
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1072): use_browser_cache:true
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1074): browser_cache_path:/tmp/mozilla-profile/cache2/
FFF: DEBUG: 2023-09-07 19:12:53,931: browsercache_firefox2.py(45): Using FirefoxCache2
FFF: DEBUG: 2023-09-07 19:12:53,931: configurable.py(1092): use_basic_cache:true
You have include_images enabled, but Python Image Library(PIL) isn't found.
Images will be included full size in original format.
Continue? (y/n)?
Y
FFF: DEBUG: 2023-09-07 19:12:56,444: adapter_fanfictionnet.py(111): URL: https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,444: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: base_browsercache.py(85): Cache Key:O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(86): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(90): /tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
FFF: DEBUG: 2023-09-07 19:12:56,445: base_browsercache.py(85): Cache Key::https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(86): :https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(90): /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
FFF: DEBUG: 2023-09-07 19:12:56,445: cache_browser.py(88): 
========== MISS (GET) BrowserCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
Traceback (most recent call last):
  File "/home/matej/.local/bin/fanficfare", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 344, in main
    dispatch(options, urls, passed_defaultsini, passed_personalini, warn, fail)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 320, in dispatch
    do_download(url,
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 435, in do_download
    adapter.getStoryMetadataOnly()
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 113, in doExtractChapterUrlsAndMetadata
    data = self.get_request(url)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 119, in get_request
    return self.get_request_redirected(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 111, in get_request_redirected
    (data,rurl) = self.configuration.get_fetcher().get_request_redirected(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/base_fetcher.py", line 133, in get_request_redirected
    fetchresp = self.do_request('GET',
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 68, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_basic.py", line 122, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 102, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_browser.py", line 97, in fetcher_do_request
    raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/12213201/1/Beautiful-Mind'
firefox-test:firefox $ ls /tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
ls: cannot access '/tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE': No such file or directory
firefox-test:firefox $  /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
bash: /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6: No such file or directory
firefox-test:firefox $ 

Using Firefox (org.mozilla.firefox) version 115.0.3 from the official Mozilla Flatpak. It seems I have an access to all those directories, but FanFicFare indicates a wrong file there.

@JimmXinu
Copy link
Owner

JimmXinu commented Aug 5, 2023

Everything you've shown here indicates that FFF looked for the appropriate cache files--the cache file names are hashed versions of the cache key which comes from the URL. They are the same for me.

But FFF isn't finding those files and your ls listing confirms the 99 file isn't there.

This commonly happens if that's the first ffnet URL you've hit recently and received the Cloudflare screen in between. That changes/doesn't cache the page. Try manually clicking to a different chapter and then back with the buttons on the page--not the browser back button--or clicking to the author page and clicking the story URL there.

@mcepl
Copy link
Contributor Author

mcepl commented Aug 7, 2023

It doesn’t like it. I have went back and forth in the story couple of time, even with Ctrl-Shift-R on the first page to eliminate a possibly bad cache and:

fun~$ ff_cache --debug https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark
-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,823: cli.py(230):     OS Version:Linux-6.4.6-1-default-x86_64-with-glibc2.37
FFF: DEBUG: 2023-08-07 07:13:39,823: cli.py(231): Python Version:3.11.4 (main, Jun 28 2023, 19:51:46) [GCC]
FFF: DEBUG: 2023-08-07 07:13:39,823: cli.py(232):    FFF Version:4.24.0
FFF: DEBUG: 2023-08-07 07:13:39,834: configurable.py(1039): use_nsapa_proxy:true
FFF: DEBUG: 2023-08-07 07:13:39,834: configurable.py(1055): use_browser_cache:true
FFF: DEBUG: 2023-08-07 07:13:39,835: configurable.py(1057): browser_cache_path:/home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2
FFF: DEBUG: 2023-08-07 07:13:39,848: browsercache_firefox2.py(45): Using FirefoxCache2
FFF: DEBUG: 2023-08-07 07:13:39,848: configurable.py(1075): use_basic_cache:true
FFF: DEBUG: 2023-08-07 07:13:39,858: adapter_fanfictionnet.py(111): URL: https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: base_browsercache.py(85): Cache Key:O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: browsercache_firefox2.py(86): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/3CF896C035993DEE2F444EA9C10DC7608AB7FC2A
FFF: DEBUG: 2023-08-07 07:13:39,858: base_browsercache.py(85): Cache Key::https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: browsercache_firefox2.py(86): :https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
FFF: DEBUG: 2023-08-07 07:13:39,858: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/D8B8996676821456783DB50D5E6A9386CF503F28
FFF: DEBUG: 2023-08-07 07:13:39,859: cache_browser.py(88): 
========== MISS (GET) BrowserCache
https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute
Traceback (most recent call last):
  File "/usr/bin/fanficfare", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/cli.py", line 344, in main
    dispatch(options, urls, passed_defaultsini, passed_personalini, warn, fail)
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/cli.py", line 320, in dispatch
    do_download(url,
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/cli.py", line 435, in do_download
    adapter.getStoryMetadataOnly()
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 113, in doExtractChapterUrlsAndMetadata
    data = self.get_request(url)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/requestable.py", line 119, in get_request
    return self.get_request_redirected(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/requestable.py", line 111, in get_request_redirected
    (data,rurl) = self.configuration.get_fetcher().get_request_redirected(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/fetchers/base_fetcher.py", line 133, in get_request_redirected
    fetchresp = self.do_request('GET',
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 68, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/fetchers/cache_basic.py", line 122, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 102, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/lib/python3.11/site-packages/fanficfare/fetchers/cache_browser.py", line 97, in fetcher_do_request
    raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/3815485/1/Harry-Potter-and-The-Trademark-Dispute'
fun~$ ls /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-rel
ease/cache2/entries/D8B8*
fish: No matches for wildcard '/home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/D8B8*'. See `help wildcards-globbing`.
ls /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/D8B8*
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
fun~$ 

@JimmXinu
Copy link
Owner

JimmXinu commented Aug 7, 2023

I can't think of anything I can do to help with the browser not caching the file.

Does it work better with Chrome? Non-flatpak Firefox? I use Firefox on win10 all the time.

I don't have time this week for more esoteric approaches, like investigating possible changes/additions to the cache hashing algorithm.

@JimmXinu
Copy link
Owner

I haven't found anything about Firefox changing their cache hash. I don't think I have anything for you.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 5, 2023

Does it work better with Chrome? Non-flatpak Firefox? I use Firefox on win10 all the time.

I don’t use Chrome and it works perfectly with non-flatpak Firefox. I am just on the MicroOS (a minimal system with almost no own packages installed, everything in Flatpaks/distrobox). I will investigate further.

Please, don’t close, this is certainly a problem.

@JimmXinu
Copy link
Owner

JimmXinu commented Sep 5, 2023

I can leave it open for now, but I really don't think this is an FFF issue.

One idea that occurred to me: are you using a VPN or proxy of some kind? We've seen various weird things happen with those that all go away without them.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 5, 2023

I can leave it open for now, but I really don't think this is an FFF issue.

One idea that occurred to me: are you using a VPN or proxy of some kind? We've seen various weird things happen with those that all go away without them.

No, nothing like that, wifi to home router connected to VDSL provider. I will try to ask around Flatpak Firefox.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 5, 2023

https://support.mozilla.org/cs/questions/1423106 let's see what happens.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 7, 2023

I can reproduce even with Firefox v117 from the mozilla tarball and FanFicFare 4.27.0:

firefox-test:firefox $ fanficfare -o use_browser_cache=true -o use_browser_cache_only=true -
o browser_cache_path=/tmp/mozilla-profile/cache2/ --debug 'https://www.fanfiction.net/s/1221
3201/1/Beautiful-Mind'
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(230):     OS Version:Linux-6.4.12-1-default-x86_64-with-glibc2.38
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(231): Python Version:3.11.4 (main, Jun 28 2023, 19:51:46) [GCC]
FFF: DEBUG: 2023-09-07 19:12:53,923: cli.py(232):    FFF Version:4.27.0
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1056): use_nsapa_proxy:true
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1072): use_browser_cache:true
FFF: DEBUG: 2023-09-07 19:12:53,930: configurable.py(1074): browser_cache_path:/tmp/mozilla-profile/cache2/
FFF: DEBUG: 2023-09-07 19:12:53,931: browsercache_firefox2.py(45): Using FirefoxCache2
FFF: DEBUG: 2023-09-07 19:12:53,931: configurable.py(1092): use_basic_cache:true
You have include_images enabled, but Python Image Library(PIL) isn't found.
Images will be included full size in original format.
Continue? (y/n)?
Y
FFF: DEBUG: 2023-09-07 19:12:56,444: adapter_fanfictionnet.py(111): URL: https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,444: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: base_browsercache.py(85): Cache Key:O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(86): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(90): /tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
FFF: DEBUG: 2023-09-07 19:12:56,445: base_browsercache.py(85): Cache Key::https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(86): :https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-07 19:12:56,445: browsercache_firefox2.py(90): /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
FFF: DEBUG: 2023-09-07 19:12:56,445: cache_browser.py(88): 
========== MISS (GET) BrowserCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
Traceback (most recent call last):
  File "/home/matej/.local/bin/fanficfare", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 344, in main
    dispatch(options, urls, passed_defaultsini, passed_personalini, warn, fail)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 320, in dispatch
    do_download(url,
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 435, in do_download
    adapter.getStoryMetadataOnly()
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 113, in doExtractChapterUrlsAndMetadata
    data = self.get_request(url)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 119, in get_request
    return self.get_request_redirected(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 111, in get_request_redirected
    (data,rurl) = self.configuration.get_fetcher().get_request_redirected(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/base_fetcher.py", line 133, in get_request_redirected
    fetchresp = self.do_request('GET',
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 68, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_basic.py", line 122, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 102, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_browser.py", line 97, in fetcher_do_request
    raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/12213201/1/Beautiful-Mind'
firefox-test:firefox $ ls /tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
ls: cannot access '/tmp/mozilla-profile/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE': No such file or directory
firefox-test:firefox $  /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
bash: /tmp/mozilla-profile/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6: No such file or directory
firefox-test:firefox $ 

@JimmXinu
Copy link
Owner

Still working for me with that story with the same cache file name.

You could try adding open_pages_in_browser:true so FFF tries opening the page for you in case it's a issue with URL mismatch.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

https://paste.opensuse.org/pastes/5b7dfc5489f2

And nothing opens in a browser.

And you still haven’t explained to me why this function provides incorrect result (from the Reddit post):

 def make_key_path(self,key):
     logger.debug(key)
     hashkey = hashlib.sha1(key.encode('utf8')).hexdigest().upper()
     fullkey = os.path.join(self.cache_dir, 'entries', hashkey)
     return fullkey

with key O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind it produces fullkey with value /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE which however, doesn’t exist on the disk.

then it tries key :https://www.fanfiction.net/s/12213201/1/Beautiful-Mind gets fullkey with value /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6 and it doesn’t exist either.

Is it possible that being Flatpaked the algorithm for locating the cache entry somehow changed?

@JimmXinu
Copy link
Owner

And you still haven’t explained to me why this function provides incorrect result (from the Reddit post):

I don't read reddit, I haven't seen whatever post you are referring to. We've previously established I can't explain your issue.

Is it possible that being Flatpaked the algorithm for locating the cache entry somehow changed?

Seems unlikely to me it would change the actual hash algorithm. IMO, it's more likely it caches in a different location, caches fewer files, or changes the key input changing the O^partitionKey part or something.

You can try using this version of browsercache_firefox2.py (unzip it, replace existing) that will scan your cache dir and output all entries <4 hours old containing the string fanfiction.net.

Using that, I see in my cache a file with an additional element after O^partitionKey that looks it may be a timestamp. But that's in addition to the existing O^partitionKey key. Could be a migration to a new key.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

Does this give you any new information?

https://paste.opensuse.org/pastes/6dc54e889391

@JimmXinu
Copy link
Owner

scandir didn't find any cache files at all with fanfiction.net in the last 4 hours.

That suggests one or more of:

  1. Your browser isn't caching in the configured browser_cache_path. Or possibly at all.
  2. Your cache file timestamps are all older than 4 hours. You could try setting browser_cache_age_limit:-1
    Do you have some weird filesystem that might timestamp files to a different timezone or something?

I'm assuming that you know how to load the story chapters in cache correctly.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

Ehhh, … I am sorry, I missed that four hours limit. https://paste.opensuse.org/pastes/175ff08832ed is after I have refreshed the cache.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

https://paste.opensuse.org/pastes/2ec711f0af38 this is with browser_cache_age_limit:-1.

@JimmXinu
Copy link
Owner

Your cache entries have an additional parameter userContextId=1& in the key. Searching for that suggests it may come from Security Containers (Firefox Multi-Account Containers add-on), do you use that?

Because after installing that, I can get cache entries with userContextId= params, too.

First, can you open a tab without a container to load the story?

If that doesn't work, this version adds userContextId=1& in the key. It works for me with tabs using the first configured container: browsercache_firefox2.zip

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

https://paste.opensuse.org/pastes/fc93e68e219c with the page opened outside of any container.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 13, 2023

Your cache entries have an additional parameter userContextId=1& in the key. Searching for that suggests it may come from Security Containers (Firefox Multi-Account Containers add-on), do you use that?

Because after installing that, I can get cache entries with userContextId= params, too.

First, can you open a tab without a container to load the story?

If that doesn't work, this version adds userContextId=1& in the key. It works for me with tabs using the first configured container: browsercache_firefox2.zip

https://paste.opensuse.org/pastes/019bc7e7d9fb

@JimmXinu
Copy link
Owner

Now I am seeing the timestamp appearing in the cache key in your cache entries:

FFF: DEBUG: 2023-09-13 22:46:11,562: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694610771,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->758E8A123FC60DE198059F3B6ACD17D00F2AED51
and
FFF: DEBUG: 2023-09-13 22:52:06,209: browsercache_firefox2.py(61): O^userContextId=1&partitionKey=%28https%2Cfanfiction.net%29,~1694610785,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->16EED09B1EAD73A49FB0D3CE6E4C053227F550C5

So adding just the userContextId=1& is insufficient, even though it would have been judging by the cache dump you gave prior.

Some (admittedly brief) further experiments seem to indicate that the timestamped cache entry appears when CloudFlare intervenes. But if you load the page directly, by clicking in the address bar and hitting enter to load the page directly or clicking the author link then the story link on the author page(since it's a single chapter story), the cache page FFF expects is saved.

@JimmXinu
Copy link
Owner

From what I've seen, I believe this should work for you, if you:

  1. Revert to the current version of browsercache_firefox2.py, or the version posted above(browsercache_firefox2.py) that also dumps recent cache.
  2. Don't use a Container tab
  3. Reload the chapter page and author page in cache directly, ie, requesting directly again when Cloudflare intervenes.

I've added some additional verbiage to the Browser Cache wiki page about reloading the page correctly:

Note: If you see a Cloudflare page before you see a story chapter page (usually when you first go to the site), that page is not loaded in cache correctly. You need to load it again by clicking in the address bar and hitting Enter (remove ?__cf... you may see), or by clicking to a different chapter/page and clicking a link back (not your browser back button). You may also need to hit Ctrl-R or click the reload button--if you see a 'Resend data' dialog on reload, you aren't on the direct page yet.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 14, 2023

I am sorry, but no.

You can see that I have the page outside of any container:
screenshot-2023-09-14_17-09-1694707167

I have reloaded couple of time, ask Firefox to remove all information about the site, certainly no CloudFlare screen was seen anywhere around, and yet:

fun:~ $ ff_cache --debug https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,708: cli.py(230):     OS Version:Linux-6.5.2-1-default-x86_64-with-glibc2.38
FFF: DEBUG: 2023-09-14 17:59:40,709: cli.py(231): Python Version:3.11.5 (main, Aug 31 2023, 07:57:41) [GCC]
FFF: DEBUG: 2023-09-14 17:59:40,709: cli.py(232):    FFF Version:4.27.0
FFF: DEBUG: 2023-09-14 17:59:40,716: configurable.py(1056): use_nsapa_proxy:true
FFF: DEBUG: 2023-09-14 17:59:40,716: configurable.py(1072): use_browser_cache:true
FFF: DEBUG: 2023-09-14 17:59:40,716: configurable.py(1074): browser_cache_path:/home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2
FFF: DEBUG: 2023-09-14 17:59:40,717: browsercache_firefox2.py(45): Using FirefoxCache2
FFF: DEBUG: 2023-09-14 17:59:40,717: configurable.py(1092): use_basic_cache:true
FFF: DEBUG: 2023-09-14 17:59:40,724: adapter_fanfictionnet.py(111): URL: https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: base_browsercache.py(85): Cache Key:O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: browsercache_firefox2.py(86): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
FFF: DEBUG: 2023-09-14 17:59:40,724: base_browsercache.py(85): Cache Key::https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: browsercache_firefox2.py(86): :https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 17:59:40,724: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
FFF: DEBUG: 2023-09-14 17:59:40,724: cache_browser.py(88): 
========== MISS (GET) BrowserCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
Traceback (most recent call last):
  File "/home/matej/.local/bin/fanficfare", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 344, in main
    dispatch(options, urls, passed_defaultsini, passed_personalini, warn, fail)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 320, in dispatch
    do_download(url,
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 435, in do_download
    adapter.getStoryMetadataOnly()
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 113, in doExtractChapterUrlsAndMetadata
    data = self.get_request(url)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 119, in get_request
    return self.get_request_redirected(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 111, in get_request_redirected
    (data,rurl) = self.configuration.get_fetcher().get_request_redirected(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/base_fetcher.py", line 133, in get_request_redirected
    fetchresp = self.do_request('GET',
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 68, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_basic.py", line 122, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 102, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_browser.py", line 97, in fetcher_do_request
    raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/12213201/1/Beautiful-Mind'
fun:~ $ 

@mcepl
Copy link
Contributor Author

mcepl commented Sep 14, 2023

With browsercache_firefox2.py from #985 (comment):

fun~$ ff_cache --debug -o browser_cache_age_limit=-1 https://www.fanfiction.net/s/12213201/1
/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,438: cli.py(230):     OS Version:Linux-6.5.2-1-default-x86_64-with-glibc2.38
FFF: DEBUG: 2023-09-14 18:03:56,438: cli.py(231): Python Version:3.11.5 (main, Aug 31 2023, 07:57:41) [GCC]
FFF: DEBUG: 2023-09-14 18:03:56,438: cli.py(232):    FFF Version:4.27.0
FFF: DEBUG: 2023-09-14 18:03:56,446: configurable.py(1056): use_nsapa_proxy:true
FFF: DEBUG: 2023-09-14 18:03:56,446: configurable.py(1072): use_browser_cache:true
FFF: DEBUG: 2023-09-14 18:03:56,446: configurable.py(1074): browser_cache_path:/home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2
FFF: DEBUG: 2023-09-14 18:03:56,447: browsercache_firefox2.py(45): Using FirefoxCache2
FFF: DEBUG: 2023-09-14 18:03:56,447: browsercache_firefox2.py(55): using scandir
FFF: DEBUG: 2023-09-14 18:03:56,447: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694671615,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->42CCE1F5E77688240B692F4556D765E3715F1CB5
FFF: DEBUG: 2023-09-14 18:03:56,447: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/cdn-cgi/challenge-platform/h/g/scripts/jsd/0cecfec7/main.js->1D899214EDA0DC87098248F3F2067F44109CECED
FFF: DEBUG: 2023-09-14 18:03:56,447: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/scripts/jquery.validatefull.20130301.js?v=1.0.0->032115EBF01D6477872C23B151856B217467DE70
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/scripts/secure_master20_min.js?v=1.0.0->6C3D772CCF66717D3D821A0087E1D3676C02E5B5
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.google.com/recaptcha/api.js->EC2981AE59C5878A027E11E2C23BEFC5E81FA4BA
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/fp.png->20569F9E28084E6AB04DEC7177F247BA01EF0012
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/twitter.png->66019B75357AFD096CE972CF4D0B0BC7C6A3C17B
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/facebook.png->C3317EB54B571C3E152E19CB820418543871A53A
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/amazon.png->5597A64D7C641F8A6493F0FD85BAB784B97DAD0F
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/msft.png->1B53C744C587DF754E9B93463A59ADA25411522F
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/social-24/google.png->D4D9D878B43994E053D9041498B6F5DC1E44399D
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,a,:https://www.gstatic.com/recaptcha/releases/0hCdE87LyjzAkFO5Ff-v7Hj1/recaptcha__en.js->AF135061ABA5C315DA121C7B210A37DEC3972E3E
FFF: DEBUG: 2023-09-14 18:03:56,448: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.gstatic.com/recaptcha/releases/0hCdE87LyjzAkFO5Ff-v7Hj1/styles__ltr.css->E42F227E408B441A932F97DD3658A8486BC3D359
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.gstatic.com/recaptcha/releases/0hCdE87LyjzAkFO5Ff-v7Hj1/recaptcha__en.js->AF4C5A58DED9BD1C097FC8B2765C5290E32E58C7
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,a,:https://fonts.gstatic.com/s/roboto/v18/KFOmCnqEu92Fr1Mu4mxK.woff2->FDC1DCC2BA224B69B18D762AA022F5688E703F8E
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.google.com/recaptcha/api2/webworker.js?hl=en&v=0hCdE87LyjzAkFO5Ff-v7Hj1->093A3C9971685070874D4271B051534143404923
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.gstatic.com/recaptcha/api2/logo_48.png->3C8BAB587192C1AB396D20F2CD3C39D0E6D39DD8
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://api-cdn.amazon.com/sdk/login1.js->0B69BD08563C08AE4D9688F5DCB2B65D9E101D76
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/cdn-cgi/challenge-platform/scripts/jsd/main.js->CF59C1BE5B3968325E60DA54694C4EBA336639BE
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js->3300482EBCC068A150AB7E279B5EAC1FA06B3938
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/api/login_state_callback.php?state=b27f3cb3cf906c1037a48fb6982b985c36de81076e2bdc8c32481790df17db5c->2AAD127DDCAB1378DEAA12E9183B101A539F88F2
FFF: DEBUG: 2023-09-14 18:03:56,449: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694671621,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->CF744780833AF29E36A7B2E7E70CDD290E3A1290
FFF: DEBUG: 2023-09-14 18:03:56,450: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694671623,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->132F51A4DAE49BA709C683E73238EC85DFF1EB36
FFF: DEBUG: 2023-09-14 18:03:56,450: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694671629,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->3165517BA7AF8F7C1842F82A8817C6C8ED845C93
FFF: DEBUG: 2023-09-14 18:03:56,450: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,~1694671643,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind->E941C41CB0F0D5E6A3B70B4337C4FED2F26A3F1A
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/styles/xss26.css?v=1.0.0->CFF05544B9A04F12525966A5A90A25C0241C6350
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/image/3086238/75/->38DA224D0A487199234125B7BB6DFD8F77592D7D
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/scripts/combo5.js?v=1.0.0->E27C2A2D2F2C99142F4C8DEE90368CCBD8B66292
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/images/d_60_90.jpg->ED5272DB0644EE7E282F3139F13EA8DD756FE143
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ff77.b-cdn.net/static/scripts/jquery.jstore_01_09_2012.js->6D2DB84D7EA41487403445B2F5F515F59932315C
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/cors/fontello-f1bf7dee/font/fontello.woff?57225533->C90C085096574A16F50BAA901677AC0C3F82EAA8
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://ajax.googleapis.com/ajax/libs/webfont/1/webfont.js->A048B95F7E62BD188B9702AA28F6851334EF75E0
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/icons3/ff-icon-180.png->2E41BCFAB51B1577488D650BB75C8C43A26FF772
FFF: DEBUG: 2023-09-14 18:03:56,451: browsercache_firefox2.py(61): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/static/icons3/ff-icon-192.png->B475BF2AE00C02E683066D946174DC473CFC85E7
FFF: DEBUG: 2023-09-14 18:03:56,452: configurable.py(1092): use_basic_cache:true
FFF: DEBUG: 2023-09-14 18:03:56,458: adapter_fanfictionnet.py(111): URL: https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: base_browsercache.py(85): Cache Key:O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: browsercache_firefox2.py(86): O^partitionKey=%28https%2Cfanfiction.net%29,:https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/34877B5340FCC2226302D53E171492C0F5FC25EE
FFF: DEBUG: 2023-09-14 18:03:56,459: base_browsercache.py(85): Cache Key::https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: browsercache_firefox2.py(86): :https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
FFF: DEBUG: 2023-09-14 18:03:56,459: browsercache_firefox2.py(90): /home/matej/.var/app/org.mozilla.firefox/cache/mozilla/firefox/g8r0rybx.default-release/cache2/entries/E3CC2B8ACDF924670604832553D0354A89CACBD6
FFF: DEBUG: 2023-09-14 18:03:56,459: cache_browser.py(88): 
========== MISS (GET) BrowserCache
https://www.fanfiction.net/s/12213201/1/Beautiful-Mind
Traceback (most recent call last):
  File "/home/matej/.bin/fanficfare", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 344, in main
    dispatch(options, urls, passed_defaultsini, passed_personalini, warn, fail)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 320, in dispatch
    do_download(url,
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/cli.py", line 435, in do_download
    adapter.getStoryMetadataOnly()
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 113, in doExtractChapterUrlsAndMetadata
    data = self.get_request(url)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 119, in get_request
    return self.get_request_redirected(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/requestable.py", line 111, in get_request_redirected
    (data,rurl) = self.configuration.get_fetcher().get_request_redirected(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/base_fetcher.py", line 133, in get_request_redirected
    fetchresp = self.do_request('GET',
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 68, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_basic.py", line 122, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/decorators.py", line 102, in fetcher_do_request
    fetchresp = chainfn(
                ^^^^^^^^
  File "/home/matej/.local/pipx/venvs/fanficfare/lib64/python3.11/site-packages/fanficfare/fetchers/cache_browser.py", line 97, in fetcher_do_request
    raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/12213201/1/Beautiful-Mind'
fun~$ 

@JimmXinu
Copy link
Owner

Okay, then it's just not going to work with that browser for now.

The question is whether it's worthwhile to re-implement cache scanning and indexing for this one particular browser sub type.

@Mikaela
Copy link

Mikaela commented Sep 14, 2023

I don't know anything about flatpaked Firefox, but are you using privacy.firstparty.isolate;true in about:config?

I noticed some time ago that Firefox outside of Flatpak doesn't function with FanFicFare unless it's disabled and Firefox restarted. I didn't report it as I mostly had the option out of habit and I am under impression that enhanced tracking protection is effective enough anyway while incompatible with that option.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 14, 2023

Actually, it started to work (probably some caches get replaced), so I was able to refresh plenty of my stories. THANK YOU!!!

Except, https://www.fanfiction.net/s/4043066/1/520-Things-I-am-not- … I guess that not complete name confused something:

fanficfare-log.txt

(sorry about too long list of cached files)

@JimmXinu
Copy link
Owner

In that case, I'm inclined to put this all down to container tabs and cache loading--which can sometimes require reloading/ctrl-shift-reloading pages and sometimes still doesn't work for unknown reasons.

@mcepl
Copy link
Contributor Author

mcepl commented Sep 14, 2023

I don't know anything about flatpaked Firefox, but are you using privacy.firstparty.isolate;true in about:config?

Actually not, I have it set to false.

@JimmXinu JimmXinu closed this as not planned Won't fix, can't repro, duplicate, stale Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants