Test partial read write

.github/workflows/testing.yml → .github/workflows/test.yml

@@ -1,5 +1,5 @@
-run-name: Run Tests
-on: [push]
+name: Tests
+on: [push, pull_request]
 jobs:
 
 
@@ -10,7 +10,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
 
     steps:
       #----------------------------------------------
@@ -49,5 +49,5 @@ jobs:
       #---- Tests
       #----------------------------------------------
 
-      - name: Run tests
-        run: poetry run python testing/run_tests.py
+      - name: 🚀 Run tests with code coverage report
+        run: poetry run pytest --cov=dictdatabase --cov-report term-missing

.gitignore

@@ -1,5 +1,7 @@
 .venv/
 .ddb_storage_testing/
+.ddb_pytest_storage
+ddb_storage
 test_db/
 *.prof
 dist/

DictDataBase.code-workspace

@@ -4,5 +4,7 @@
 			"path": "."
 		}
 	],
-	"settings": {}
-}
+	"settings": {
+		"python.pythonPath": ".venv/bin/python3"
+	}
+}

README.md

@@ -1,106 +1,174 @@
 # DictDataBase
 
+
+
 [![Downloads](https://pepy.tech/badge/dictdatabase)](https://pepy.tech/project/dictdatabase)
+
 [![Downloads](https://pepy.tech/badge/dictdatabase/month)](https://pepy.tech/project/dictdatabase)
+
 [![Downloads](https://pepy.tech/badge/dictdatabase/week)](https://pepy.tech/project/dictdatabase)
 
-DictDataBase is a simple but fast and secure database for handling dicts (or PathDicts for more advanced features), that uses json files as the underlying storage mechanism.
-It is also multiprocessind and multithreading safe, due to the employed locking mechanisms.
+DictDataBase is a simple but fast and secure database for handling dicts (or PathDicts for more advanced features), that uses json or compressed json as the underlying storage mechanism. It is:
+- **Multi threading and multi processing safe**. Multiple processes on the same machine can simultaneously read and write to dicts without writes getting lost.
+- **No database server** required. Simply import DictDataBase in your project and use it.
+- **ACID** compliant. Unlike TinyDB, it is suited for concurrent environments.
+- **Fast**. A dict can be accessed partially without having to parse the entire file, making the read and writes very efficient.
+- **Tested** with over 400 test cases.
 
-## Import
+### Why use DictDataBase
+- For example have a webserver dispatches database read and writes concurrently.
+- If spinning up a database server is overkill for your app.
+	- But you still need [ACID](https://en.wikipedia.org/wiki/ACID) guarantees
+- You have a big database, only want to access one key-value pair. DictDataBase can do this efficiently and fast.
+- Your use case is suited for working with json data, or you have to work with a lot of json data.
 
-```python
-import DictDataBase as DDB
-```
+### Why not DictDataBase
+- If you need document indexes
+- If your use case is better suited for a sql database
 
 
-## Configuration
+# Configuration
+There are 5 configuration options:
 
-There are 3 configuration options.
+### Storage directory
 Set storage_directory to the path of the directory that will contain your database files:
-
 ```python
+
 DDB.config.storage_directory = "./ddb_storage" # Default value
 ```
 
+### Compression
 If you want to use compressed files, set use_compression to True.
-This will make the db files significantly smaller and might improve performance if your disk is slow.
-However, the files will not be human readable.
+This will make the db files significantly smaller and might improve performance if your disk is slow. However, the files will not be human readable.
 ```python
+
 DDB.config.use_compression = False # Default value
+
 ```
 
-If you set pretty_json_files to True, the json db files will be indented and the keys will be sorted.
-It won't affect compressed files, since the are not human-readable anyways.
+### Indentation
+Set the way how written json files should be indented. Behaves exactly like json.dumps(indent=...). It can be an `int` for the number of spaces, the tab character, or `None` if you don't want the files to be indented.
 ```python
-DDB.config.pretty_json_files = True # Default value
+
+DDB.config.indent = "\t" # Default value
+
 ```
 
 
-You can specify your own json encoder and decoder if you need to.
+### Sort keys
+Specify if you want the dict keys to be sorted when writing to a file.Behaves exactly like json.dumps(sort_keys=...).
+```python
+
+DDB.config.sort_keys = True # Default value
+
+```
+
+### Use orjson
+You can specify the orjson encoder and decoder if you need to.
 The standard library json module is sufficient most of the time.
-However, alternatives like orjson might be more performant for your use case.
-The encoder function should take a dict and return a str or bytes.
-The decoder function should take a string and return a dict.
+However, orjson is a lot more performant in virtually all cases.
 ```python
-DDB.config.custom_json_encoder = None # Default value
-DDB.config.custom_json_decoder = None # Default value
+
+DDB.config.use_orjson = True # Default value
+
 ```
 
 
+# Usage
 
+## Import
 
-## Create dicts
-Before you can access dicts, you need to explicitly create them.
+```python
 
-Do create ones that already exist, this would raise an exception.
-Also do not access ones that do not exist, this will also raise an exception.
+import DictDataBase as DDB
 
+```
+
+
+## Create dict
+This library is called DictDataBase, but you can actually use any json serializable object.
 ```python
+
 user_data_dict = {
 	"users": {
-		"Ben": {
-			"age": 30,
-			"job": "Software Engineer"
-		},
-		"Sue": {
-			"age": 21:
-			"job": "Student"
-		},
-		"Joe": {
-			"age": 50,
-			"job": "Influencer"
-		}
+		"Ben": { "age": 30, "job": "Software Engineer" },
+		"Sue": { "age": 21, "job": "Student" },
+		"Joe": { "age": 50, "job": "Influencer" }
 	},
 	"follows": [["Ben", "Sue"], ["Joe", "Ben"]]
 })
-DDB.create("user_data", db=user_data_dict)
-# There is now a file called user_data.json (or user_data.ddb if you use compression)
+
+DDB.at("user_data").create(user_data_dict)
+
+# There is now a file called user_data.json
+# (or user_data.ddb if you use compression)
 # in your specified storage directory.
 ```
 
+## Check if exists
+
+
 
 ## Read dicts
+
 ```python
-d = DDB.read("user_data")
+
+d = DDB.at("user_data").read()
 # You now have a copy of the dict named "user_data"
 print(d == user_data_dict) # True
+
+
+# Only partially read Joe
+joe = DDB.at("user_data").read("Joe")
+print(joe == user_data_dict["Joe"])
+
 ```
 
-## Write dicts
 
+## Write dicts
 ```python
+
 import DictDataBase as DDB
-with DDB.session("user_data") as (session, user_data):
-	# You now have a handle on the dict named "user_data"
-	# Inside the with statement, the file of user_data will be locked, and no other
-	# processes will be able to interfere.
-	user_data["follows"].append(["Sue", "Ben"])
-	session.write()
-	# Now the changes to d are written to the database
-
-print(DDB.read("user_data")["follows"])
+
+with DDB.at("user_data").session() as (session, user_data):
+
+# You now have a handle on the dict named "user_data"
+
+# Inside the with statement, the file of user_data will be locked, and no other
+
+# processes will be able to interfere.
+
+user_data["follows"].append(["Sue", "Ben"])
+
+session.write()
+
+# Now the changes to d are written to the database
+
+
+
+print(DDB.at("user_data").read()["follows"])
+
 # -> [["Ben", "Sue"], ["Joe", "Ben"], ["Sue", "Ben"]]
+
 ```
 
 If you do not call session.write(), the database file will not be modified.
+
+
+# API Reference
+
+### at()
+
+## DDBMethodChooser
+
+### exists()
+
+### haskey() (can also be part of exists)
+
+### create()
+
+### delete()
+
+### read()
+
+### session()

cli.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+while [ $# -gt 0 ]; do case $1 in
+
+
+  --test|-t)
+    poetry run pytest --cov=dictdatabase --cov-report term-missing
+    rm ./.coverage
+    shift ;;
+
+
+
+  *|-*|--*)
+    echo "Unknown option $1"
+    echo "Usage: [ -t | --test ] [ -p | --profiler ]"
+    exit 2
+    exit 1 ;;
+
+
+esac; done

dictdatabase/__init__.py

@@ -1,3 +1 @@
-from . reading import exists, read, multiread
-from . writing import create, delete, session, multisession
-from . models import SubModel
+from . models import SubModel, at

dictdatabase/config.py

@@ -1,7 +1,5 @@
-from typing import Optional, Callable
-
 storage_directory = "./ddb_storage"
 use_compression = False
-pretty_json_files = True
-custom_json_encoder: Optional[Callable[[dict], str | bytes]] = None
-custom_json_decoder: Optional[Callable[[str], dict]] = None
+use_orjson = True
+indent = "\t"  # eg. "\t" or 4 or None
+sort_keys = True

dictdatabase/io_safe.py

@@ -1,6 +1,5 @@
 import os
-from . locking import ReadLock, WriteLock
-from . import config, utils, io_unsafe
+from . import config, utils, io_unsafe, locking
 
 
 def read(db_name: str):
@@ -14,10 +13,27 @@ def read(db_name: str):
 	if not json_exists and not ddb_exists:
 		return None
 	# Wait in any write lock case, "need" or "has".
-	lock = ReadLock(db_name)
-	res = io_unsafe.read(db_name)
-	lock.unlock()
-	return res
+	lock = locking.ReadLock(db_name)
+	try:
+		return io_unsafe.read(db_name)
+	except BaseException as e:
+		raise e
+	finally:
+		lock.unlock()
+
+
+def partial_read(db_name: str, key: str):
+	_, json_exists, _, ddb_exists = utils.db_paths(db_name)
+	if not json_exists and not ddb_exists:
+		return None
+	# Wait in any write lock case, "need" or "has".
+	lock = locking.ReadLock(db_name)
+	try:
+		return io_unsafe.partial_read(db_name, key).key_value
+	except BaseException as e:
+		raise e
+	finally:
+		lock.unlock()
 
 
 def write(db_name: str, db: dict):
@@ -26,10 +42,13 @@ def write(db_name: str, db: dict):
 	"""
 	dirname = os.path.dirname(f"{config.storage_directory}/{db_name}.any")
 	os.makedirs(dirname, exist_ok=True)
-
-	write_lock = WriteLock(db_name)
-	io_unsafe.write(db_name, db)
-	write_lock.unlock()
+	write_lock = locking.WriteLock(db_name)
+	try:
+		io_unsafe.write(db_name, db)
+	except BaseException as e:
+		raise e
+	finally:
+		write_lock.unlock()
 
 
 def delete(db_name: str):
@@ -39,9 +58,13 @@ def delete(db_name: str):
 	json_path, json_exists, ddb_path, ddb_exists = utils.db_paths(db_name)
 	if not json_exists and not ddb_exists:
 		return None
-	write_lock = WriteLock(db_name)
-	if json_exists:
-		os.remove(json_path)
-	if ddb_exists:
-		os.remove(ddb_path)
-	write_lock.unlock()
+	write_lock = locking.WriteLock(db_name)
+	try:
+		if json_exists:
+			os.remove(json_path)
+		if ddb_exists:
+			os.remove(ddb_path)
+	except BaseException as e:
+		raise e
+	finally:
+		write_lock.unlock()