[fantia] Question about fantia scraping and extractor configuration #4068

etui-null · 2023-05-16T10:52:37Z

etui-null
May 16, 2023

I was looking to start archiving a few fantia accounts I'm subscribed to but couldn't figure out a way to emulate the folder structure I've been manually using as well as how to scrape all the text I was interested in. On fantia, creators can pay wall specific sections of a post to different tiers. I like to keep each of these sections contained in their own folder; however, when I checked the keywords available for a link I couldn't find anything that would allow me to break up each section.

I tried to find a good example of this from some random post on the front page and this should illustrate what I'm talking about.
Post: https://fantia.jp/posts/1964477, word of warning, this link is NSFW. It's downright impossible to find SFW links on fantia.

> gallery-dl -K "https://fantia.jp/posts/1964477"
Keywords for directory names:
-----------------------------
category
  fantia
comment
  🐮見放題プランの方向けに３年前の投稿を公開していきます🐮
いつも支援して頂きありがとうございます！
For Unlimited plan subscribers, posts from three years ago are now available! Thank you for your support!



⇩🐧下スクロールでエロ差分🐧⇩

date
  2023-05-14 09:00:00
fanclub_id
  2931
fanclub_name
  🐧軒下の猫屋🐧
fanclub_url
  https://fantia.jp/fanclubs/2931
fanclub_user_id
  86032
fanclub_user_name
  アルデヒド
post_id
  1964477
post_title
  【見放題プラン】グラブルなまあし部　エウロペさん
post_url
  https://fantia.jp/posts/1964477
posted_at
  Sun, 14 May 2023 18:00:00 +0900
rating
  adult
subcategory
  post
tags[N]['name']
  グラブル
tags[N]['uri']
  /fanclubs/2931/posts?tag=%E3%82%B0%E3%83%A9%E3%83%96%E3%83%AB

Keywords for filenames and --filter:
------------------------------------
category
  fantia
comment
  🐮見放題プランの方向けに３年前の投稿を公開していきます🐮
いつも支援して頂きありがとうございます！
For Unlimited plan subscribers, posts from three years ago are now available! Thank you for your support!



⇩🐧下スクロールでエロ差分🐧⇩

content_category
  thumb
content_filename
  
date
  2023-05-14 09:00:00
extension
  png
fanclub_id
  2931
fanclub_name
  🐧軒下の猫屋🐧
fanclub_url
  https://fantia.jp/fanclubs/2931
fanclub_user_id
  86032
fanclub_user_name
  アルデヒド
file_id
  thumb
file_url
  https://c.fantia.jp/uploads/post/file/1964477/48a60e5f-1487-4063-b952-1e0e87b49c3d.png
filename
  48a60e5f-1487-4063-b952-1e0e87b49c3d
num
  1
post_id
  1964477
post_title
  【見放題プラン】グラブルなまあし部　エウロペさん
post_url
  https://fantia.jp/posts/1964477
posted_at
  Sun, 14 May 2023 18:00:00 +0900
rating
  adult
subcategory
  post
tags[N]['name']
  グラブル
tags[N]['uri']
  /fanclubs/2931/posts?tag=%E3%82%B0%E3%83%A9%E3%83%96%E3%83%AB

And here's an image of what the post sections look like.

Each section can have a title as well as additional comment information, should any be written. If all of this has already been implemented, could someone help me build the extractor config and postprocessor details needed to scrape the contents of the tiers into dedicated folders as well as write a text file to that folder with any relevant info?

Answered by mikf

May 24, 2023

There is an old, still open issue asking for the same feature: #2477

With the way gallery-dl currently works regarding directory creation, it would not be possible to create such a directory structure, even if this data was available.

View full answer

etui-null · 2023-05-24T05:55:50Z

etui-null
May 24, 2023
Author

Still looking for any help on this issue. Though I did make a few discoviers that should probably give someone more knowledgeable about config building all the info they'd need to help. I found an old issue #2381 that talks about the "Blog Post" format with a great included test post. https://fantia.jp/posts/1166373

Originally I tried testing with the following postprocessor configs:
Config 1:

            "postprocessors": 
            [{
                "name": "metadata",
                "event": "post",
                "filename": "{post_title}.txt",
            }]

Config 2:

            "postprocessors": 
            [{
                "name": "metadata",
                "event": "post",
                "filename": "{post_title}.txt",

                "#": "write text content",
                "format": [
                    "{content:?//}",
                    "{html:?//}",
                    "{text:?//}",
                    "{excerpt:?//}"
                ]
            }]

But in each case basically nothing of interest to me was written.
Run with config 1:

{
    "post_id": 1166373,
    "post_url": "https://fantia.jp/posts/1166373",
    "post_title": "Test Fantia Post",
    "comment": "\n\n",
    "rating": "general",
    "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
    "date": "2022-03-09 16:46:12",
    "fanclub_id": 356320,
    "fanclub_user_id": 7487131,
    "fanclub_user_name": "2022/03/08 15:13:52の名無し",
    "fanclub_name": "Test Fantia",
    "fanclub_url": "https://fantia.jp/fanclubs/356320",
    "tags": [],
    "category": "fantia",
    "subcategory": "post"
}

Run with config 2 provided an empty file (with some new line characters that would have broken up each section).

But then I have found the command line flag -j which lets you dump the json file for whatever page is being loaded, and that seems to have the info I'm looking for. You can see the json for the above link below. Reading through the output, it looks like every section is contained in each of the numbered elements. Inside each of these are definitions for comments and titles. All of this is the info I'm looking to ideally break into a directory tree, but if that's not possible just having all of that get written to a meta data file (any of the content titles and comments). I just don't know how to access it and write a config/metadata post processor to make use of it. If anyone could help guide me in the right direction I'd really appreciate it.

Just to be clear, the directory tree I'm looking to replicate looks like this:

12345-Fantia name
└── 67890-This is the main title
    ├── Free tier
    │   ├── 0.jpeg
    │   ├── 1.jpeg
    │   ├── 2.jpeg
    │   ├── 3.jpeg
    │   └── section_text.txt
    ├── $3 tier
    │   ├── 0.jpeg
    │   ├── 1.jpeg
    │   ├── 2.jpeg
    │   ├── 3.jpeg
    │   └── section_text.txt
    ├── $5 tier
    │   ├── 0.jpeg
    │   ├── 1.jpeg
    │   ├── 2.jpeg
    │   ├── 3.jpeg
    │   └── section_text.txt
    ├── post.jpeg
    └── post_text.txt

Json dump for the above test post.

[
  [
    2,
    {
      "category": "fantia",
      "comment": "\n\n",
      "date": "2022-03-09 16:46:12",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://c.fantia.jp/uploads/post/file/1166373/7e7e7fda-720d-462d-ab9b-a4c558ff5780.png",
    {
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "thumb",
      "content_filename": "",
      "date": "2022-03-09 16:46:12",
      "extension": "png",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": "thumb",
      "file_url": "https://c.fantia.jp/uploads/post/file/1166373/7e7e7fda-720d-462d-ab9b-a4c558ff5780.png",
      "filename": "7e7e7fda-720d-462d-ab9b-a4c558ff5780",
      "num": 1,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://cc.fantia.jp/uploads/post_content_photo/file/7325490/5b8138fe-83e3-41d5-b282-33857e65683e.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS83MzI1NDkwLzViODEzOGZlLTgzZTMtNDFkNS1iMjgyLTMzODU3ZTY1NjgzZS5wbmciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODQ5MDcyMTZ9fX1dfQ__&Signature=2QwbxLmMBcVmF0D2vdnTjl7~~-PJEK3fQMNUD0bcxtz5qdqZVuIOcZgX17dwYtNViCZIfP4gQAUptQYrsc6lGANdgGCQ9sbU1Hc~8mRUHWWoodYCcHLDiH7fZhJjt3iXpkWeIO14yvyo-aaeAK5Z~sUJ~i6y7mOTQsxBOGPU7lrdLW293MPM8uUbRoic2W4fbVGtv4g57LH9aKs1EVOhqiHL2t6BvZWizypzmo1SsBqkr0Z4K5M52ndC~5rG9~R-GkRpCgMqYK6nRsA18KOrRv8TKahkBxo7NU80p27dM862Jv33BtU-QmByIExRfnqsgolC12D7GqV61bpZOwCFiw__",
    {
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "photo_gallery",
      "content_comment": "This is an image gallery.",
      "content_filename": "",
      "content_id": 1870739,
      "content_title": "Test Image Gallery",
      "date": "2022-03-09 16:46:12",
      "extension": "png",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": 7325490,
      "file_url": "https://cc.fantia.jp/uploads/post_content_photo/file/7325490/5b8138fe-83e3-41d5-b282-33857e65683e.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS83MzI1NDkwLzViODEzOGZlLTgzZTMtNDFkNS1iMjgyLTMzODU3ZTY1NjgzZS5wbmciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODQ5MDcyMTZ9fX1dfQ__&Signature=2QwbxLmMBcVmF0D2vdnTjl7~~-PJEK3fQMNUD0bcxtz5qdqZVuIOcZgX17dwYtNViCZIfP4gQAUptQYrsc6lGANdgGCQ9sbU1Hc~8mRUHWWoodYCcHLDiH7fZhJjt3iXpkWeIO14yvyo-aaeAK5Z~sUJ~i6y7mOTQsxBOGPU7lrdLW293MPM8uUbRoic2W4fbVGtv4g57LH9aKs1EVOhqiHL2t6BvZWizypzmo1SsBqkr0Z4K5M52ndC~5rG9~R-GkRpCgMqYK6nRsA18KOrRv8TKahkBxo7NU80p27dM862Jv33BtU-QmByIExRfnqsgolC12D7GqV61bpZOwCFiw__",
      "filename": "5b8138fe-83e3-41d5-b282-33857e65683e",
      "num": 2,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://cc.fantia.jp/uploads/post_content_photo/file/7325491/5f708327-2035-4085-ad8c-c08bdcb6ce57.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS83MzI1NDkxLzVmNzA4MzI3LTIwMzUtNDA4NS1hZDhjLWMwOGJkY2I2Y2U1Ny5wbmciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODQ5MDcyMTZ9fX1dfQ__&Signature=wlb8MSImIUUZN0SUurp0I69Ui-GD-RB14opFkDNpPjOr7OhDaP0OWVepf~q~hEhLben127FZQnw0eDk5CGP4i~EbNHUTGd7buXmeKxcXBEm89PQ6scYE~PRU6x6BT9jn1yBdAddeVhgg~SUyW2rAj5qf7DHkNPTfRQDbrPQd3iJmfUMp7DN8iuiwRjWXuLM8EJ9KWNFqwuzQk9uELDW8lGruTRV0EC2p2YTSZzFJSOZJh1PlVjzudVmKqOss3lV6-avYG17gbvrc~OOzWsEa9QJ-Ec7XkQQP0ufhhkz5p2XVjJfwsCYBmbPJTW4XMgQa7LSSEinxeTQYc1gJJo5fwg__",
    {
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "photo_gallery",
      "content_comment": "This is an image gallery.",
      "content_filename": "",
      "content_id": 1870739,
      "content_title": "Test Image Gallery",
      "date": "2022-03-09 16:46:12",
      "extension": "png",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": 7325491,
      "file_url": "https://cc.fantia.jp/uploads/post_content_photo/file/7325491/5f708327-2035-4085-ad8c-c08bdcb6ce57.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS83MzI1NDkxLzVmNzA4MzI3LTIwMzUtNDA4NS1hZDhjLWMwOGJkY2I2Y2U1Ny5wbmciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODQ5MDcyMTZ9fX1dfQ__&Signature=wlb8MSImIUUZN0SUurp0I69Ui-GD-RB14opFkDNpPjOr7OhDaP0OWVepf~q~hEhLben127FZQnw0eDk5CGP4i~EbNHUTGd7buXmeKxcXBEm89PQ6scYE~PRU6x6BT9jn1yBdAddeVhgg~SUyW2rAj5qf7DHkNPTfRQDbrPQd3iJmfUMp7DN8iuiwRjWXuLM8EJ9KWNFqwuzQk9uELDW8lGruTRV0EC2p2YTSZzFJSOZJh1PlVjzudVmKqOss3lV6-avYG17gbvrc~OOzWsEa9QJ-Ec7XkQQP0ufhhkz5p2XVjJfwsCYBmbPJTW4XMgQa7LSSEinxeTQYc1gJJo5fwg__",
      "filename": "5f708327-2035-4085-ad8c-c08bdcb6ce57",
      "num": 3,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://fantia.jp/posts/1166373/album_image?query=NIJDJv9e%2FD9j0fqt6pqzq%2BocgMV72NaHzM5adeBKX09dFg2mlAw0Ppg5pJKGwaUEsVnXTV2STCsFUrnCkekauDsAzz%2B71c0KY37tq2VwMVZfCzH2YvZr--is1k%2BUd1SfRYwKTU--8wTd0bPRllae8ZMLXGUrLw%3D%3D",
    {
      "blogpost_text": "This is a test.\n\nThis is a test.\n\n",
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "blog",
      "content_comment": "{\"ops\":[{\"insert\":\"This is a test.\\n\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130683\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130683/5d243250-1acb-45b5-80da-5f4904f1ddc4.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY4My81ZDI0MzI1MC0xYWNiLTQ1YjUtODBkYS01ZjQ5MDRmMWRkYzQucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE2fX19XX0_\\u0026Signature=dg478Y5P4eMdWrKae0HRYr1D3mRxTYndjagKM8g7oYVA8OrhpF5fg2KQePxW0L8juLM~7sRk8VqWjpK6hTTPwkI3xTyCBNHdQypD6OBc15UrmPzmBJf4S6DGC4SiKciimfHe~nzBWu9hqncK2gxnRitvfoaZwbnmAx9g1LO9-BESBIXhURbQV3QCj1vQ2W6VafJ~xAW2IdNSwOBvJOoYnkRR~IgOrP3keI8zp2kMmKVKEJDq-rPBk64qwZhvTfiJPEvwtzGL9CLTs9eYUQSMkOLTvEOd6zVdXCYih~47gp22b4Ul6epvmqZ9P8IbMrxLbhhA3RbiK73NZwQCHl8sfA__\",\"original_url\":\"/posts/1166373/album_image?query=NIJDJv9e%2FD9j0fqt6pqzq%2BocgMV72NaHzM5adeBKX09dFg2mlAw0Ppg5pJKGwaUEsVnXTV2STCsFUrnCkekauDsAzz%2B71c0KY37tq2VwMVZfCzH2YvZr--is1k%2BUd1SfRYwKTU--8wTd0bPRllae8ZMLXGUrLw%3D%3D\"}}},{\"insert\":\"This is a test.\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130684\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130684/870c6d07-a105-4e04-bfd1-42f75bdc1746.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY4NC84NzBjNmQwNy1hMTA1LTRlMDQtYmZkMS00MmY3NWJkYzE3NDYucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE2fX19XX0_\\u0026Signature=Y-kHHeYO8eZRqR8VOSNwcBNvh3MXLwBsJNRWkdgr0A0yYsluQW14bdnsOjAJ8NS5qs8Z8mbtMZO~XbCqs5kKjysarw13KrELs1lsYpNICNuKgURdQTQhciK~L7gxr5yOwstLbTKL8SsqZqFzfmKqI0zkAmxQ-CXmZI9LyP-ZkfEf~2Plc-C2RZUK0Gn-nqgzcTj9cgT~Juz2KdBDAjIP-nnOPeAt3GD1IwjR5Nek2wv1kH0yOXf7u~jCUEVAz5GLS-eU8XDf7g8-vCN80St1fK3MdaSR6zWTA-tB1yKLapGtkM-C-b4R5qGmzSfE2qIYKuHDcl5Gqjh7plRobcHewA__\",\"original_url\":\"/posts/1166373/album_image?query=z3LWnrdaRKyyfGBJqoUXZ%2BM5nVwAtBfxCp4qz1dNUVxGTyoj4tLR%2FMMPilG6z%2FlA7tA1iflxZ3Ixs8kDCXC7mBsq4LI8sWP7t9QbDYWdPiMP1j35acEj--F%2FV7Ym5BUSfGilaF--qmKYpKvHviu8P%2BhbYHFL7A%3D%3D\"}}},{\"insert\":\"\\n\"}]}",
      "content_filename": "",
      "content_id": 1870740,
      "content_title": "Test Blog Content 1",
      "date": "2022-03-09 16:46:12",
      "extension": "",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": "130683",
      "file_url": "https://fantia.jp/posts/1166373/album_image?query=NIJDJv9e%2FD9j0fqt6pqzq%2BocgMV72NaHzM5adeBKX09dFg2mlAw0Ppg5pJKGwaUEsVnXTV2STCsFUrnCkekauDsAzz%2B71c0KY37tq2VwMVZfCzH2YvZr--is1k%2BUd1SfRYwKTU--8wTd0bPRllae8ZMLXGUrLw%3D%3D",
      "filename": "album_image",
      "num": 4,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://fantia.jp/posts/1166373/album_image?query=z3LWnrdaRKyyfGBJqoUXZ%2BM5nVwAtBfxCp4qz1dNUVxGTyoj4tLR%2FMMPilG6z%2FlA7tA1iflxZ3Ixs8kDCXC7mBsq4LI8sWP7t9QbDYWdPiMP1j35acEj--F%2FV7Ym5BUSfGilaF--qmKYpKvHviu8P%2BhbYHFL7A%3D%3D",
    {
      "blogpost_text": "This is a test.\n\nThis is a test.\n\n",
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "blog",
      "content_comment": "{\"ops\":[{\"insert\":\"This is a test.\\n\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130683\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130683/5d243250-1acb-45b5-80da-5f4904f1ddc4.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY4My81ZDI0MzI1MC0xYWNiLTQ1YjUtODBkYS01ZjQ5MDRmMWRkYzQucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE2fX19XX0_\\u0026Signature=dg478Y5P4eMdWrKae0HRYr1D3mRxTYndjagKM8g7oYVA8OrhpF5fg2KQePxW0L8juLM~7sRk8VqWjpK6hTTPwkI3xTyCBNHdQypD6OBc15UrmPzmBJf4S6DGC4SiKciimfHe~nzBWu9hqncK2gxnRitvfoaZwbnmAx9g1LO9-BESBIXhURbQV3QCj1vQ2W6VafJ~xAW2IdNSwOBvJOoYnkRR~IgOrP3keI8zp2kMmKVKEJDq-rPBk64qwZhvTfiJPEvwtzGL9CLTs9eYUQSMkOLTvEOd6zVdXCYih~47gp22b4Ul6epvmqZ9P8IbMrxLbhhA3RbiK73NZwQCHl8sfA__\",\"original_url\":\"/posts/1166373/album_image?query=NIJDJv9e%2FD9j0fqt6pqzq%2BocgMV72NaHzM5adeBKX09dFg2mlAw0Ppg5pJKGwaUEsVnXTV2STCsFUrnCkekauDsAzz%2B71c0KY37tq2VwMVZfCzH2YvZr--is1k%2BUd1SfRYwKTU--8wTd0bPRllae8ZMLXGUrLw%3D%3D\"}}},{\"insert\":\"This is a test.\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130684\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130684/870c6d07-a105-4e04-bfd1-42f75bdc1746.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY4NC84NzBjNmQwNy1hMTA1LTRlMDQtYmZkMS00MmY3NWJkYzE3NDYucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE2fX19XX0_\\u0026Signature=Y-kHHeYO8eZRqR8VOSNwcBNvh3MXLwBsJNRWkdgr0A0yYsluQW14bdnsOjAJ8NS5qs8Z8mbtMZO~XbCqs5kKjysarw13KrELs1lsYpNICNuKgURdQTQhciK~L7gxr5yOwstLbTKL8SsqZqFzfmKqI0zkAmxQ-CXmZI9LyP-ZkfEf~2Plc-C2RZUK0Gn-nqgzcTj9cgT~Juz2KdBDAjIP-nnOPeAt3GD1IwjR5Nek2wv1kH0yOXf7u~jCUEVAz5GLS-eU8XDf7g8-vCN80St1fK3MdaSR6zWTA-tB1yKLapGtkM-C-b4R5qGmzSfE2qIYKuHDcl5Gqjh7plRobcHewA__\",\"original_url\":\"/posts/1166373/album_image?query=z3LWnrdaRKyyfGBJqoUXZ%2BM5nVwAtBfxCp4qz1dNUVxGTyoj4tLR%2FMMPilG6z%2FlA7tA1iflxZ3Ixs8kDCXC7mBsq4LI8sWP7t9QbDYWdPiMP1j35acEj--F%2FV7Ym5BUSfGilaF--qmKYpKvHviu8P%2BhbYHFL7A%3D%3D\"}}},{\"insert\":\"\\n\"}]}",
      "content_filename": "",
      "content_id": 1870740,
      "content_title": "Test Blog Content 1",
      "date": "2022-03-09 16:46:12",
      "extension": "",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": "130684",
      "file_url": "https://fantia.jp/posts/1166373/album_image?query=z3LWnrdaRKyyfGBJqoUXZ%2BM5nVwAtBfxCp4qz1dNUVxGTyoj4tLR%2FMMPilG6z%2FlA7tA1iflxZ3Ixs8kDCXC7mBsq4LI8sWP7t9QbDYWdPiMP1j35acEj--F%2FV7Ym5BUSfGilaF--qmKYpKvHviu8P%2BhbYHFL7A%3D%3D",
      "filename": "album_image",
      "num": 5,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://fantia.jp/posts/1166373/album_image?query=26saOL0QnhRcFFTRBz4FuFdugr9%2FFqcLgxve39L%2F0oGuyVm2O7ip0%2FdedyyXi4DCe%2FiN3pS1z2Iq4fWqx7yZghDjggRpZI%2B6yoIj0qcE2QAUbCentn6u--SGEsVeXXYDjR8%2FyM--CXYXhxBh%2B4l5fMypKFIYaw%3D%3D",
    {
      "blogpost_text": "This is a test.\n\n\n\n",
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "blog",
      "content_comment": "{\"ops\":[{\"insert\":\"This is a test.\\n\\n\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130685\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130685/db615f84-4fca-4bb9-81a9-47c217866c56.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY4NS9kYjYxNWY4NC00ZmNhLTRiYjktODFhOS00N2MyMTc4NjZjNTYucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE3fX19XX0_\\u0026Signature=mgWFJjygq3Mypj3Mygax3PW0Nrzr8RJ2auKVIdC5w13RNujYbS0NYg~9tVBCrHWX54EtfAS1z2VEj8o7LMsr5BDNUa4Ton8DvqgAqy6pAaMrudoyiWP3XkCIB8bUuBc4Qc5TMd-AFfZaB3l84~B~tdwVckgVwdKxjqHFgo170LASwu6jBNunfp6mRHGS5GhLyp73p0i9QtyIjFCuuCPJloNfcQRVjYigcP~YU8Rwq1-nVq~4KnzPwLrudcYc1OOl0~mNaFyXn2iHTOKN1quwvu7vJbhtjKaF0MDIzDVlqmZEVFfmO9qBhwGDHNqH6cx5a~5KBHRPADhwt0f9fBSZxg__\",\"original_url\":\"/posts/1166373/album_image?query=26saOL0QnhRcFFTRBz4FuFdugr9%2FFqcLgxve39L%2F0oGuyVm2O7ip0%2FdedyyXi4DCe%2FiN3pS1z2Iq4fWqx7yZghDjggRpZI%2B6yoIj0qcE2QAUbCentn6u--SGEsVeXXYDjR8%2FyM--CXYXhxBh%2B4l5fMypKFIYaw%3D%3D\"}}},{\"insert\":\"\\n\"}]}",
      "content_filename": "",
      "content_id": 1870741,
      "content_title": "Test Blog Content 2",
      "date": "2022-03-09 16:46:12",
      "extension": "",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": "130685",
      "file_url": "https://fantia.jp/posts/1166373/album_image?query=26saOL0QnhRcFFTRBz4FuFdugr9%2FFqcLgxve39L%2F0oGuyVm2O7ip0%2FdedyyXi4DCe%2FiN3pS1z2Iq4fWqx7yZghDjggRpZI%2B6yoIj0qcE2QAUbCentn6u--SGEsVeXXYDjR8%2FyM--CXYXhxBh%2B4l5fMypKFIYaw%3D%3D",
      "filename": "album_image",
      "num": 6,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ],
  [
    3,
    "https://fantia.jp/posts/1166373/album_image?query=JVtEpc1hYna7O75%2Bdpaplwo3obgfLHw3%2BybiNs%2FP9sKD3ZmFabpDRgfyD%2BWIgmW7M465wQ0GFPDYuMKH9CIl9OszqnvUTU%2FU%2BdBb5v79esDvAovvM2kB--nJg6xmhvAKE2l%2F62--DyMdam03mrQ8bN42foXWww%3D%3D",
    {
      "blogpost_text": "Link to video:\nhttps://www.youtube.com/watch?v=5SSdvNcAagI\n\nhtml img from another site:\n\n\n\n\n\n",
      "category": "fantia",
      "comment": "\n\n",
      "content_category": "blog",
      "content_comment": "{\"ops\":[{\"insert\":\"Link to video:\\n\"},{\"attributes\":{\"link\":\"https://www.youtube.com/watch?v=5SSdvNcAagI\"},\"insert\":\"https://www.youtube.com/watch?v=5SSdvNcAagI\"},{\"insert\":\"\\n\\nhtml img from another site:\\n\"},{\"insert\":{\"image\":\"https://www.w3schools.com/images/lamp.jpg\"}},{\"insert\":\"\\n\\n\\n\\n\"},{\"insert\":{\"fantiaImage\":{\"id\":\"130693\",\"url\":\"https://cc.fantia.jp/uploads/album_image/file/130693/687b20d8-b30d-4373-a322-124478710ada.png?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A\\u0026Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9hbGJ1bV9pbWFnZS9maWxlLzEzMDY5My82ODdiMjBkOC1iMzBkLTQzNzMtYTMyMi0xMjQ0Nzg3MTBhZGEucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjg0OTA3MjE3fX19XX0_\\u0026Signature=Gferx~Y2JJqEVhyGfsWMLn8jA8NWirnfni4b54fgsQfXc-Z2J71x5~wHD2-t2XxDU8dEyU52M~zo40YqOjmrucQ6d49n7lJnxYZOgnlGEAh72972TrFmyhYcoqNw64zveQozbCBFUktBWt-x3Qoj7VJfmZJGPoI8cqQxpr6uweliNh413Rk-KNFBVFVTrn9eF6d0TGlL~GvFPy-SkYc6zPwNGjfvY4UEf464GZNFTsUTEQ4CJ6ew7y~LbBANw7KKDLzaRuVKftefRvXuA6tBJhW3RIRMdSiU~SOFTsBJeY5V4rDwSajWh0pEaAU~wPp0beHRDG1XMB4jFie6Ljwcmg__\",\"original_url\":\"/posts/1166373/album_image?query=JVtEpc1hYna7O75%2Bdpaplwo3obgfLHw3%2BybiNs%2FP9sKD3ZmFabpDRgfyD%2BWIgmW7M465wQ0GFPDYuMKH9CIl9OszqnvUTU%2FU%2BdBb5v79esDvAovvM2kB--nJg6xmhvAKE2l%2F62--DyMdam03mrQ8bN42foXWww%3D%3D\"}}},{\"insert\":\"\\n\"}]}",
      "content_filename": "",
      "content_id": 1870870,
      "content_title": "Test Blog Content 3 (Links & HTML Importing?)",
      "date": "2022-03-09 16:46:12",
      "extension": "",
      "fanclub_id": 356320,
      "fanclub_name": "Test Fantia",
      "fanclub_url": "https://fantia.jp/fanclubs/356320",
      "fanclub_user_id": 7487131,
      "fanclub_user_name": "2022/03/08 15:13:52\u306e\u540d\u7121\u3057",
      "file_id": "130693",
      "file_url": "https://fantia.jp/posts/1166373/album_image?query=JVtEpc1hYna7O75%2Bdpaplwo3obgfLHw3%2BybiNs%2FP9sKD3ZmFabpDRgfyD%2BWIgmW7M465wQ0GFPDYuMKH9CIl9OszqnvUTU%2FU%2BdBb5v79esDvAovvM2kB--nJg6xmhvAKE2l%2F62--DyMdam03mrQ8bN42foXWww%3D%3D",
      "filename": "album_image",
      "num": 7,
      "post_id": 1166373,
      "post_title": "Test Fantia Post",
      "post_url": "https://fantia.jp/posts/1166373",
      "posted_at": "Thu, 10 Mar 2022 01:46:12 +0900",
      "rating": "general",
      "subcategory": "post",
      "tags": []
    }
  ]
]

0 replies

mikf · 2023-05-24T10:43:04Z

mikf
May 24, 2023
Maintainer

There is an old, still open issue asking for the same feature: #2477

With the way gallery-dl currently works regarding directory creation, it would not be possible to create such a directory structure, even if this data was available.

6 replies

mikf May 24, 2023
Maintainer

There is not. When there are no metadata etc options listed in docs/configuration, then what you see with -j is all that is available.

You could take a look at the files created with --write-pages to see the raw data fetched by gallery-dl. Maybe there's something in there that's currently ignored.

etui-null May 24, 2023
Author

Gotcha, thanks for the information!

akeboshiKOU Jun 1, 2023

@etui-null I've included some relevant code from the fetched raw data file of the Fantia test post referenced here, in #4128.
Seems like info regarding tiers is present after all, if made available as keywords might prove to be useful in setting up your directory tree!

etui-null Jun 3, 2023
Author

@akeboshiKOU That's a great dump! Thanks a lot for the info!

mikf Mar 23, 2024
Maintainer

The mentioned open issue got resolved some time ago. You can now sort files by plan[…].

Charlotte-br560 · 2024-03-23T07:29:11Z

Charlotte-br560
Mar 23, 2024

For scraping content from different tiers of a Fantia post and organizing them into dedicated folders, a custom extractor configuration is needed due to the lack of granularity in existing keywords. You'll also require a postprocessor to organize the extracted content and relevant information into folders and text files respectively. Crawlbase could be a useful tool for this task.

1 reply

mikf Mar 23, 2024
Maintainer

Same bot as #4497 (comment).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fantia] Question about fantia scraping and extractor configuration #4068

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[fantia] Question about fantia scraping and extractor configuration #4068

etui-null May 16, 2023

Replies: 3 comments · 7 replies

etui-null May 24, 2023 Author

mikf May 24, 2023 Maintainer

mikf May 24, 2023 Maintainer

etui-null May 24, 2023 Author

akeboshiKOU Jun 1, 2023

etui-null Jun 3, 2023 Author

mikf Mar 23, 2024 Maintainer

Charlotte-br560 Mar 23, 2024

mikf Mar 23, 2024 Maintainer

etui-null
May 16, 2023

Replies: 3 comments 7 replies

etui-null
May 24, 2023
Author

mikf
May 24, 2023
Maintainer

mikf May 24, 2023
Maintainer

etui-null May 24, 2023
Author

etui-null Jun 3, 2023
Author

mikf Mar 23, 2024
Maintainer

Charlotte-br560
Mar 23, 2024

mikf Mar 23, 2024
Maintainer