BaseCrawler – A Modern Desktop Database Management Tool

"A powerful, lightweight GUI that lets you explore, edit, and query relational and document databases — all from a single window."

License GitHub Tag GitHub Language Count

Table of Contents

Section

1. Overview
2. Features
3. Prerequisites & Installation
4. Getting Started
5. Interface Guide
6. Connection Management
7. Architecture
8. Contributing
9. Roadmap
10. License

1. Overview

BaseCrawler is a desktop GUI database management tool built with Python and CustomTkinter. It lets you:

• Connect to and manage multiple PostgreSQL, MySQL, MariaDB, and MongoDB databases simultaneously.
• Browse, create, edit, and delete tables and collections — all without writing SQL by hand.
• View, add, edit, and delete rows from an inline table viewer with persistent column sizing and primary key indicators.
• Run ad-hoc SQL queries against any connected database from a dedicated query window.
• Save connection profiles with optional password storage, and instantly switch between them without losing your place.

BaseCrawler is written in Python 3.10+ and uses CustomTkinter for a native-feeling dark/light mode UI on macOS, Windows, and Linux.

2. Features

Category	Feature
Connections	Saved connection profiles stored in connections.json Supports PostgreSQL, MySQL, MariaDB, and MongoDB Background status polling — live Online / Offline indicator per connection Optional password saving per connection with "Save password" toggle Per-connection workspace state — switch connections and return to exactly where you left off One-click Disconnect from the ribbon toolbar
Schema Browser	Table / collection list with single-click select and double-click open Add Table wizard with per-column PK, Auto, Rand, Length, and Unique options Edit Table dialog — modify types, add/drop columns, change constraints Delete Table with confirmation dialog MongoDB: collections shown in list; DDL operations automatically hidden 5-second auto-refresh keeps the schema list current
Inline Table Viewer	Full-width inline data view — no separate popup windows 🔑 icon on primary key column headers Alternating row colors with a toggleable grid-line overlay Resizable columns — widths persist across refreshes per session Double-click a column separator to auto-fit to content Double-click any cell for inline editing Add Row with auto-fill for identity and default columns Edit Row and Delete Row with confirmation Add Column and Delete Column while table is open 5-second auto-refresh keeps data current
Query Wizard	Non-modal query window — stays open while you browse the schema SQL editor with Execute and Clear actions Formatted results (column headers + rows) displayed inline Auto-refreshes the schema list after any DDL statement (CREATE, ALTER, DROP, etc.) Accessible from both the schema view and the table view via the ribbon
Ribbon Toolbar	MS Office-style ribbon with labeled icon groups Schema view groups: View · Tables · Query · Connection Table view groups: Navigate · Rows · Columns · Query Edit Mode toggle (✏) — dims structural editing buttons when off Grid Lines toggle (⊞) — show/hide alternating row colors
UI & Workspace	Dark and Light mode via CustomTkinter Collapsible connection sidebar (◀ Hide Panel / ▶ Show Panel) Per-connection open-table state restored on reconnect All database calls run on background threads — UI stays responsive

3. Prerequisites & Installation

Required:

• Python 3.10+
• A base-crawler conda environment (recommended) or a plain virtual environment

Using conda (recommended)

git clone https://github.com/Netherwarlord/BaseCrawler.git
cd BaseCrawler
conda create -n base-crawler python=3.11
conda activate base-crawler
pip install -r requirements.txt

Using a plain virtual environment

git clone https://github.com/Netherwarlord/BaseCrawler.git
cd BaseCrawler
python -m venv .venv
source .venv/bin/activate   # macOS / Linux
.venv\Scripts\activate      # Windows
pip install -r requirements.txt

macOS – if pip complains about the system Python

brew link --force --overwrite python

Key dependencies

Package	Purpose
customtkinter	Modern themed GUI framework
CTkMessagebox	Styled dialog boxes
psycopg2	PostgreSQL driver
mysql-connector-python	MySQL / MariaDB driver
pymongo	MongoDB driver

4. Getting Started

Launch the application

Always activate the conda environment first, then run app.py directly:

conda activate base-crawler
python app.py

To syntax-check without launching the GUI:

python -m py_compile app.py

Add your first connection

1. Click Manage Connections at the bottom of the left sidebar.
2. Click Add Connection and fill in the connection name, database type, host, port, username, password, and database name.
3. Click Save Connection. The connection now appears in the sidebar.

Connect and explore

1. Click a connection name in the sidebar to select it — the right panel shows connection details and a live status indicator.
2. Enter your username and password in the credentials panel (check Save password if you want it stored).
3. Click Connect. The ribbon toolbar and table list appear automatically.
4. Double-click any table in the list to open it inline, or single-click and press ▶ Open in the ribbon.

5. Interface Guide

Top Bar

The narrow bar above the workspace contains three persistent controls:

Control	Function
◀ Hide Panel / ▶ Show Panel	Collapse or expand the left connection sidebar
⊞ (checkbox)	Toggle alternating row grid lines in the table viewer
✏ (checkbox)	Toggle Edit Mode — when off, structural editing buttons are dimmed

Schema View Ribbon

Visible when browsing the table list. Groups and their icon buttons:

Group	Buttons
View	▶ Open — open the selected table inline
Tables	✏️ Edit · ➕ Add · 🗑️ Delete
Query	≡ SQL — open the Query Wizard window
Connection	⏻ Disconnect — disconnect and return to the connection info panel

Table View Ribbon

Visible when a table is open inline. Groups and their icon buttons:

Group	Buttons
Navigate	◀ Back — return to the table list
Rows	➕ Add · ✏️ Edit · 🗑️ Delete
Columns	➕ Add · 🗑️ Delete
Query	≡ SQL — open the Query Wizard window

Column Resizing

• Drag a column separator in the header to resize. Width is remembered for the rest of the session.
• Double-click a column separator to auto-fit the column to its widest content.

6. Connection Management

Connection Info Screen

When you select a saved connection (but are not yet connected), the right panel shows:

• Connection name — large, top-left
• Info card — Type, Host, Port, DB, live Status (Online / Offline / Checking…), and Size
• Credentials panel — editable Username and Password fields, pre-filled from saved values
• Save password checkbox — controls whether the password is persisted to connections.json
• Connect button — or press Enter in either credential field

Saved Connections File

Connections are stored in connections.json in the project root. This file is created automatically the first time you save a connection. Do not commit this file if it contains sensitive credentials.

Switching Connections

Click any connection in the sidebar while already connected. BaseCrawler will:

1. Save your current workspace state (which table is open, the cached schema).
2. Silently reconnect to the new connection.
3. Restore your previous workspace for that connection if one exists.

7. Architecture

BaseCrawler is three Python files with no web server or external process:

File	Responsibility
connection_manager.py	Pure data layer. Reads/writes connections.json. No UI dependency. Key methods: add_connection, remove_connection, update_connection, reorder_connection.
db_connector.py	One DBConnector base class, three concrete implementations: PostgreSQLConnector (psycopg2), MongoDBConnector (pymongo), MySQLMariaDBConnector (mysql-connector-python). get_connector(connection_details) is the factory. All connectors share a common interface: connect, disconnect, fetch_schema, fetch_data, execute_query, insert_data, update_data, delete_data, fetch_column_defaults, fetch_primary_keys, evaluate_expression.
app.py	All UI. Key classes: DBManagerApp (main window), ManageConnectionsWindow, NewConnectionWindow, AddEditDataWindow (row-level add/edit form), AddTableDialog, EditTableDialog, AddColumnDialog, DeleteColumnDialog, QueryWindow. All DB calls that run in the background use queue.Queue + after() polling — never updating CTk widgets directly from a background thread.

Threading Model

All database calls that run in the background communicate results back to the main thread via queue.Queue + after() polling — UI widgets are never updated from a background thread.

The status poller (_start_status_poller) is a long-lived daemon thread per selected connection, sleeping 30 s between checks. The auto-refresh timer (_start_auto_refresh) fires every 5 seconds via after() on the main thread, refreshing either the schema list or the open table depending on the current view.

8. Contributing

We welcome contributions! Whether you have a bug report, a feature request, or just want to improve documentation — open an issue first so we can discuss it before diving into code.

Development Setup

git clone https://github.com/Netherwarlord/BaseCrawler.git
cd BaseCrawler
conda create -n base-crawler python=3.11
conda activate base-crawler
pip install -r requirements.txt

Style & Linting

PEP 8 compliance — run flake8 locally.

Black formatter — apply with black .

10. Roadmap

Version	Description	Release Date
v1.0.0	Initial Release Full database support for: PostgreSQL, Oracle DB, MySQL, MariaDB, MongoDB and MSDB Full-Feature toolbar for manipulating databases. Support for custom modules and databse connecotrs. Modern UI and design language. Improved performance and reliability. Native support for secrets management, parameterized query enforcement, and identity-bound auditing to satisfy SOC 2 and PCI DSS requirements out of the box.	Coming Soon
v0.8.4-alpha	Minor Stability Fixes	2026-05-04
v0.8.3-alpha	Major Overhaul Separated main app.py file into smaller modules. Made performance improvements on load times. Added persistent state saving across restarts. Added double-click to edit cuntionality in active tables. Modularized Wizard design for later implementation.	2026-05-03
v0.7.51-alpha	Initial Pre-Release Alpha First alpha version fit for public testing stages. Stabilized Thread Leak issue. Created single window functionality. Added toolbar to table viewer/editor. Moved Query editor into its own window/wizard. Finalized PostgreSQL connector. Finalized MongoDB connecotr. Finalized MariaDB connector. Finalized MySQL connector.	2026-05-03

11. License

This project is licensed under the GNU Public License V3 (GPL3). See the LICENSE file for details.

Enjoy CRAWLING your data! 🚀