Skip to content

Data Import & NoSQL Queries

Jump to bottom
Lukatrum edited this page May 20, 2026 · 8 revisions

This wiki page outlines the data import methods and NoSQL querying capabilities available in JDb.

Data Import Features

The JDb object supports importing data from various formats and relational databases.

1. INI & TOML Import

You can load configurations directly from INI or TOML formats:

  • INI: Use jdb.from_ini() to parse INI data.
  • TOML: Use jdb.from_toml() to parse TOML data.

2. SQLite Import

Data from SQLite databases can be migrated using jdb.from_sqlite(db_path).

  • Each table in the SQLite database is automatically converted into a specific group within JDb.
  • For example, you can access the imported projects table using jdb.get_group('projects') or jdb['projects'].

NoSQL Query Features

The find() method provides robust, MongoDB-style NoSQL querying capabilities to filter your JSON data.

Basic & Regex Matching

  • Exact Match: Use the ANY parameter, such as jdb.find(ANY={'name': 'Alice'}) or jdb.find(ANY='Alice').
  • Regular Expressions: Use the vals or RE parameters for regex matching, such as jdb.find(vals={'name': re.compile(r'li[a-z]')}) or jdb.find(RE=r'\D30\D').

Logical & Comparison Operators

Queries support advanced dict-based operators for precise filtering:

  • Comparison: $eq (equal), $ne (not equal), $le (less than or equal to), and $ge (greater than or equal to).
  • Range & Inclusion: Use $lt (less than), $gt (greater than), and $in (to check if a value exists within a specific list).
  • Array Operators: Use $size to match the exact length of a list or array (e.g., {'tags': {'$size': 2}}).
  • Logical Operators: Combine conditions using $and, $or, and $not within the vals dictionary. Alternatively, you can use direct keyword arguments like AND=[...], OR=[...], and NOT={...} for cleaner syntax.

Lambda Queries

Use Python lambda functions directly as dictionary keys to filter data:

# Find all records where the value ends with '1'
matches = jdb[lambda key: key.endswith('1')]

# Find nested data where role is 'Developer'
devs = jdb.find(FUNC=lambda k, v: v['role'] == 'Developer')

Regex Queries (RE)

Filter keys or values using powerful Regular Expressions:

# Slicing syntax for Regex (Find keys matching regex)
matches = jdb[::r'user_\d+$']

# Find records containing 'John' or 'Bob' in their values
matches = jdb.find(RE='John|Bob')

Time-Series & Date Slicing

Every record in omni-json-db is timestamped, unlocking powerful date-based lookups.

import datetime as dt

now = dt.datetime.now()
yesterday = now - dt.timedelta(days=1)

# Get all records modified today
today_records = jdb[dt.date.today()]

# Slice by date range: records between yesterday and now
recent_records = jdb[yesterday:now]

# Records older than yesterday
old_records = jdb[:yesterday]

More Examples

Below are examples of how to utilize the various parameters and NoSQL syntax.

from omni_json_db import JDb
import re

# Initialize an in-memory database
jdb = JDb()

# Sample user records
users = [
   {'name': 'Alice', 'age': 30, 'email': 'alice@example.com', 'role': 'admin', 'tags': ['python', 'database']},
   {'name': 'Bob', 'age': 25, 'role': 'developer', 'tags': ['javascript', 'web']},
   {'name': 'Charlie', 'age': 35, 'role': 'developer', 'tags': ['python', 'linux', 'aws']},
   {'name': 'Diana', 'age': 28, 'email': 'diana@test.com', 'role': 'designer', 'tags': ['ui', 'ux']}
]

# Insert data (using auto-generated keys or iterative insertion)
for i, user in enumerate(users):
    jdb[f'user_{i}'] = user

1. Exact Match & Global Search (ANY, RE, RE2)

Find records where any field exactly matches or contains a specific value.

# Find users where any attribute exactly matches 'Alice'
res = jdb.find(ANY='Alice')

# RE/RE2 convert value into JSON string format for searching.
# Find any record that has the string 'designer' inside it
res = jdb.find(RE=r'designer')

# RE2 remove some JSON symbol (,[]{}") before searching
res = jdb.find(RE2=r'role:designer')

2. Relational & Conditional Operators (vals)

Filter data within dictionary fields using NoSQL operators ($eq, $ne, $lt, $le, $lte, $gt, $ge, $gte, $in, $nin, $has).

# Age is greater than or equal to 30
res = jdb.find(vals={'age': {'$ge': 30}})

# Age is strictly less than 30
res = jdb.find(vals={'age': {'$lt': 30}})

# Role is either 'admin' or 'designer'
res = jdb.find(vals={'role': {'$in': ['admin', 'designer']}})

# tags contains 'python'
res = jdb.find(vals={'tags': {'$has': 'python'}})

# Age is NOT 30
res = jdb.find(vals={'age': {'$ne': 30}})

# Age is 28
res = jdb.find(vals={'age': {'$eq': 28}})

3. Logical Grouping (AND, OR, NOT)

Combine multiple conditions for complex lookups.

# Age >= 25 AND Age <= 30
res = jdb.find(AND=[{'age': {'$ge': 25}}, {'age': {'$le': 30}}])

# Role is 'admin' OR Age > 30
res = jdb.find(OR=[{'role': 'admin'}, {'age': {'$gt': 30}}])

# User is NOT a developer
res = jdb.find(NOT={'role': 'developer'})

# (Role is 'admin' OR Age > 30) AND 'linux' not in tags
res = jdb.find(AND=[
   {'$or': [
      {'role': 'admin'},
      {'age': {'$gt': 30}}
   ]},
   {'$not': {'tags': {'$has': 'linux'}}}
])

4. Regular Expressions (RE, RE2, re.compile)

JDb natively supports regex for fuzzy matching on both keys and values.

# Values matching an email domain regex
res = jdb.find(vals={'email': re.compile(r'.@example.com')})

# Find users where any attribute exactly matches regex
res = jdb.find(ANY=re.compile(r'.@example.com'))

# Global regex search for strings containing 'li' (matches 'Alice', 'Charlie', 'linux')
res = jdb.find(RE=r'li[a-z]')

# Match specific Database Keys using compiled regex (e.g., matching 'user_1', 'user_2')
res = jdb.find(re.compile(r'^user_[1-2]$'))

5. Array / List Operations

Directly query list sizes or elements at specific array indices.

# Users with exactly 2 tags in their list
res = jdb.find(vals={'tags': {'$size': 2}})

# Users whose FIRST tag (index 0) is 'python'
res = jdb.find(vals={'tags': {'$0': 'python'}})

6. Lambda / Custom Functions (FUNC) & Pagination (limit)

For highly specific rules, pass a Python function. Use limit to stop searching once enough results are found.

# Pass a lambda to evaluate both the key and the value dynamically
# Example: Find the first 2 users whose age is an even number
res = jdb.find(
    FUNC=lambda k, v: isinstance(v, dict) and v.get('age', 1) % 2 == 0, 
    limit=2
)

# Users has email
res = jdb.find(vals={'email': lambda v: v != ''})

# Users don't have email
res = jdb.find(NOT={'email': lambda v: v != ''})

# For primitive stored values (non-nested), you can use quick keyword arguments:
jdb['simple_counter'] = 50
res = jdb.find(EQ=50)       # Equals 50
res = jdb.find(IN=[40, 50]) # Value in list

Clone this wiki locally