Skip to content

Type37/DA-note-scraper

Repository files navigation

DeviantArt Notes Scraper

Download all your DeviantArt notes with metadata including sender, date, subject, and more. Export to JSON or CSV format for backup and archival purposes.

πŸš€ Three Versions Available

πŸ€– Browser Scraper (RECOMMENDED for Messages)

β†’ See Browser Scraper Guide - Works with the new Messages system!

If your account has been migrated from Notes to Messages, use this version:

  • βœ… Works with Messages - scrapes directly from the website
  • βœ… No API needed - uses browser automation
  • βœ… Gets everything - full conversation history
  • ⚠️ Requires Chrome and your DA login credentials
python scraper.py

🌐 Web Version (For Notes API only)

β†’ Open Web App - Browser-based, no installation

  • βœ… Zero installation - runs in your browser
  • βœ… Cross-platform - works on any device
  • βœ… Privacy-focused - all processing happens locally
  • ❌ Notes API only - won't work if you've been migrated to Messages

🐍 Python CLI Version (For Notes API only)

Command-line tool for power users

  • βœ… Full control via command line
  • βœ… Scriptable - integrate into workflows
  • βœ… Backend authentication - uses OAuth2 with client secret
  • ❌ Notes API only - won't work if you've been migrated to Messages

See Python CLI instructions below.


Python CLI Version

Features

  • OAuth2 Authentication: Secure authentication with DeviantArt API
  • Complete Note Download: Downloads all notes from inbox and custom folders
  • Rich Metadata: Captures sender, recipient, date, subject, read status, and more
  • Multiple Export Formats: Export to JSON or CSV
  • Pagination Support: Handles large note collections automatically
  • Statistics: View summary statistics about your notes
  • User-Friendly CLI: Simple command-line interface

Installation

  1. Clone this repository:
git clone <repository-url>
cd DA-note-scraper
  1. Install dependencies:
pip install -r requirements.txt
  1. Set up DeviantArt API credentials:
cp .env.example .env
  1. Edit .env and add your credentials:
DA_CLIENT_ID=your_client_id_here
DA_CLIENT_SECRET=your_client_secret_here
DA_REDIRECT_URI=https://localhost:5000/callback

Usage

Basic Usage

Download all notes and export to JSON:

python main.py

Export Options

Export to CSV:

python main.py --format csv

Export to both JSON and CSV:

python main.py --format both

Specify custom output filename:

python main.py --output my_notes --format both

Other Options

Skip custom folders (inbox only):

python main.py --no-folders

Force re-authentication:

python main.py --reauth

Help

python main.py --help

Exported Metadata

Each note includes the following metadata:

  • note_id: Unique identifier for the note
  • folder: Folder name where the note is stored
  • subject: Note subject/title
  • body: Full note content
  • sender: Username of the sender
  • sender_user_icon: URL to sender's avatar
  • recipient: Username of the recipient
  • date: Unix timestamp
  • date_readable: Human-readable date (YYYY-MM-DD HH:MM:SS)
  • is_read: Whether the note has been read
  • is_deleted: Whether the note is deleted
  • type: Note type

Example Output

JSON Format

[
  {
    "note_id": "12345",
    "folder": "Inbox",
    "subject": "Hello!",
    "body": "Thanks for the watch!",
    "sender": "username123",
    "recipient": "your_username",
    "date": "1234567890",
    "date_readable": "2024-01-15 14:30:00",
    "is_read": true,
    "is_deleted": false,
    "type": "note"
  }
]

CSV Format

note_id,folder,subject,sender,recipient,date_readable,is_read,body
12345,Inbox,Hello!,username123,your_username,2024-01-15 14:30:00,True,Thanks for the watch!

Project Structure

DA-note-scraper/
β”œβ”€β”€ main.py              # CLI interface
β”œβ”€β”€ auth.py              # OAuth2 authentication
β”œβ”€β”€ notes_downloader.py  # Notes downloading and export logic
β”œβ”€β”€ config.py            # Configuration management
β”œβ”€β”€ requirements.txt     # Python dependencies
β”œβ”€β”€ .env.example         # Example environment variables
β”œβ”€β”€ .gitignore          # Git ignore rules
└── README.md           # This file

Important Notes

DeviantArt API Changes

DeviantArt is migrating from the Notes system to a new Messages system. This tool uses the Notes API which may be deprecated in the future. Make sure to download your notes while the API is still available.

OAuth Callback

The default OAuth callback URL is https://localhost:5000/callback. When you run the authentication flow:

  1. A browser window will open
  2. Authorize the application on DeviantArt
  3. You'll be redirected to localhost:5000
  4. The app will capture the authorization code automatically
  5. The browser will show "Authorization successful!"

Token Storage

Your access token is stored in .da_token.json (automatically added to .gitignore). This allows you to run the script multiple times without re-authenticating.

To force re-authentication, use the --reauth flag or delete the .da_token.json file.

Troubleshooting

"DA_CLIENT_ID is not set"

Make sure you've created a .env file with your DeviantArt API credentials.

Authentication fails

  1. Check that your Client ID and Secret are correct
  2. Verify the redirect URI matches what's configured in your DeviantArt app
  3. Try using --reauth to clear the old token

"Notes folders endpoint not found"

This is normal if you don't have any custom note folders. The script will continue with the inbox.

No notes downloaded

  • Make sure you're logged into the correct DeviantArt account
  • Check that you have notes in your account
  • Try using --reauth to ensure you're using the correct account

API Rate Limits

DeviantArt's API has rate limits. If you have many notes, the download may take some time. The script handles pagination automatically and will download all your notes.

Privacy & Security

  • Never commit your .env file or .da_token.json to version control
  • Your credentials are only used to authenticate with DeviantArt's official API
  • Downloaded notes are stored locally on your machine
  • No data is sent to any third-party services

License

This project is provided as-is for personal use. Make sure to comply with DeviantArt's Terms of Service and API usage policies.

Contributing

Feel free to submit issues or pull requests to improve this tool.

Sources

This tool was developed using information from:

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •