Skip to content

Gplay-scraper v1.0.2

Choose a tag to compare

View all tags
@Mohammedcha Mohammedcha released this 15 Oct 08:07
· 10 commits to main since this release
v1.0.2
e1035b0

Major Release - Complete Library Redesign 🚀

This version represents a complete rewrite of GPlay Scraper with a focus on modularity, extensibility, and comprehensive data extraction across all Google Play Store features.

New Features

7 Method Types with 42 Functions

  • App Methods - Extract 65+ data fields from any app (ratings, installs, pricing, permissions, screenshots, etc.)
  • Search Methods - Search Google Play Store apps with comprehensive filtering and pagination
  • Reviews Methods - Extract user reviews with ratings, timestamps, helpful votes, and detailed feedback
  • Developer Methods - Get all apps published by a specific developer using developer ID
  • List Methods - Access top charts (TOP_FREE, TOP_PAID, TOP_GROSSING) by category with 54 categories
  • Similar Methods - Find similar/competitor apps for market research and competitive analysis
  • Suggest Methods - Get search suggestions and autocomplete for ASO keyword research

Each method type includes 6 functions:

  • analyze() - Get all data as dictionary/list
  • get_field() - Get single field value
  • get_fields() - Get multiple fields as dictionary
  • print_field() - Print single field to console
  • print_fields() - Print multiple fields to console
  • print_all() - Print all data as formatted JSON

7 HTTP Clients with Automatic Fallback

  • requests (default) - Standard Python HTTP library, reliable and well-tested
  • curl_cffi - Browser impersonation with TLS fingerprinting, best for avoiding detection
  • tls_client - Custom TLS fingerprinting, good for bypassing restrictions
  • httpx - Modern async-capable HTTP client with HTTP/2 support
  • urllib3 - Low-level HTTP client with connection pooling
  • cloudscraper - Cloudflare bypass capabilities
  • aiohttp - Async HTTP client for high-performance concurrent requests

Automatic fallback system tries clients in order until one succeeds, ensuring maximum reliability.

Multi-Language & Multi-Region Support

  • Support for 100+ languages (en, es, fr, de, ja, ko, zh, ar, etc.)
  • Support for 150+ countries (us, gb, ca, au, in, br, jp, etc.)
  • Get localized app data, reviews, and search results
  • Region-specific pricing and availability information

Comprehensive Data Extraction

  • 65+ App Fields: title, developer, ratings, installs, price, screenshots, permissions, release date, update date, size, version, content rating, privacy policy, and more
  • Review Data: user name, rating, review text, timestamp, app version, helpful votes, developer reply
  • Search Results: app ID, title, developer, rating, price, icon, screenshots, description snippet
  • Developer Portfolio: all apps from a developer with complete metadata
  • Top Charts: ranked lists with install counts, ratings, and trending data
  • Similar Apps: competitor analysis with relevance scoring
  • Search Suggestions: popular keywords and autocomplete terms

Enhanced Architecture

  • Modular Design: Separate classes for methods, scrapers, and parsers
  • Core Modules: gplay_methods.py, gplay_scraper.py, gplay_parser.py
  • HTTP Client Abstraction: HttpClient class with pluggable client support
  • Element Specs: Reusable CSS selector specifications for data extraction
  • Helper Utilities: Text processing, date parsing, JSON cleaning, age calculation
  • Exception Hierarchy: 6 custom exception types for specific error scenarios

Documentation & Testing

  • Comprehensive Docstrings: All 42 methods, 7 scrapers, 7 parsers, and utility functions documented
  • Sphinx Documentation: Professional HTML documentation with examples, API reference, and guides
  • HTTP Clients Guide: Detailed documentation on when and how to use each HTTP client
  • Fields Reference: Complete reference of all 65+ fields, categories, and parameters
  • Unit Tests: Complete test coverage for all 7 method types
  • Examples: Real-world usage examples for each method type

Configuration & Customization

  • Configurable Parameters: Language, country, count, sort order, collection type
  • Rate Limiting: Built-in delays to prevent blocking (configurable)
  • Error Handling: Graceful fallbacks and informative error messages
  • Logging: Detailed logging for debugging and monitoring
  • Timeout Control: Configurable request timeouts
  • Retry Logic: Automatic retries with exponential backoff

Breaking Changes

  • Complete API redesign - not backward compatible with v1.0.1
  • Method names changed from get_app_details() to app_analyze()
  • New parameter structure for all methods
  • HTTP client must be specified or uses automatic fallback
  • Exception types renamed and reorganized

Migration Guide

Old (v1.0.1):

scraper = GPlayScraper()
data = scraper.get_app_details("com.whatsapp")

New (v1.0.2):

scraper = GPlayScraper()
data = scraper.app_analyze("com.whatsapp")

Performance Improvements

  • Faster JSON parsing with optimized regex patterns
  • Reduced memory usage with streaming parsers
  • Better caching of HTTP client instances
  • Parallel request support with async clients

Bug Fixes

  • Fixed JSON parsing for apps with special characters in descriptions
  • Fixed review extraction for apps with no reviews
  • Fixed developer ID extraction from developer pages
  • Fixed category parsing for apps in multiple categories
  • Fixed price parsing for apps with regional pricing
  • Fixed screenshot URL extraction for apps with video previews
Assets 2
Loading