Allow disk-based persistence to be completely disabled

.vscode/launch.json

@@ -17,6 +17,19 @@
       "skipFiles": ["node_modules/*/**", "<node_internals>/*/**"],
       "outputCapture": "std"
     },
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Azurite Service - Loki, in-memory",
+      "cwd": "${workspaceFolder}",
+      "runtimeArgs": ["-r", "ts-node/register"],
+      "args": ["${workspaceFolder}/src/azurite.ts", "-d", "debug.log", "--inMemoryPersistence"],
+      "env": {
+        "AZURITE_ACCOUNTS": ""
+      },
+      "skipFiles": ["node_modules/*/**", "<node_internals>/*/**"],
+      "outputCapture": "std"
+    },
     {
       "type": "node",
       "request": "launch",
@@ -204,6 +217,34 @@
       "internalConsoleOptions": "openOnSessionStart",
       "outputCapture": "std"
     },
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Current Mocha TS File - Loki, in-memory",
+      "cwd": "${workspaceFolder}",
+      "runtimeArgs": ["-r", "ts-node/register"],
+      "args": [
+        "${workspaceFolder}/node_modules/mocha/bin/_mocha",
+        "-u",
+        "tdd",
+        "--timeout",
+        "999999",
+        "--colors",
+        "${workspaceFolder}/${relativeFile}"
+      ],
+      "env": {
+        "AZURITE_ACCOUNTS": "",
+        "AZURE_TABLE_STORAGE": "",
+        "DATATABLES_ACCOUNT_NAME": "<name of your storage account>",
+        "DATATABLES_ACCOUNT_KEY" : "<account key for your storage account>",
+        "AZURE_DATATABLES_STORAGE_STRING": "https://<your account name>.table.core.windows.net",
+        "AZURE_DATATABLES_SAS": "?<sas query string>",
+        "NODE_TLS_REJECT_UNAUTHORIZED": "0",
+        "AZURITE_TEST_INMEMORYPERSISTENCE": "true"
+      },
+      "internalConsoleOptions": "openOnSessionStart",
+      "outputCapture": "std"
+    },
     {
       "name": "VSC Extension",
       "type": "extensionHost",

.vscodeignore

@@ -30,4 +30,5 @@ temp
 .github
 .prettierrc.json
 README.mcr.md
-release
+release
+docs

ChangeLog.md

@@ -4,6 +4,10 @@
 
 ## Upcoming Release
 
+General:
+
+- Add `--inMemoryPersistence`  and `--extentMemoryLimit` options and related configs to store all data in-memory without disk persistence. (issue #2227)
+
 ## 2023.10 Version 3.27.0
 
 General:

README.md

@@ -33,6 +33,8 @@
     - [Certificate Configuration (HTTPS)](#certificate-configuration-https)
     - [OAuth Configuration](#oauth-configuration)
     - [Skip API Version Check](#skip-api-version-check)
+    - [Disable Product Style Url](#disable-product-style-url)
+    - [Use in-memory storage](#use-in-memory-storage)
     - [Command Line Options Differences between Azurite V2](#command-line-options-differences-between-azurite-v2)
   - [Supported Environment Variable Options](#supported-environment-variable-options)
     - [Customized Storage Accounts & Keys](#customized-storage-accounts--keys)
@@ -198,6 +200,8 @@ Following extension configurations are supported:
 - `azurite.oauth` OAuth oauthentication level. Candidate level values: `basic`.
 - `azurite.skipApiVersionCheck` Skip the request API version check, by default false.
 - `azurite.disableProductStyleUrl` Force parsing storage account name from request Uri path, instead of from request Uri host.
+- `azurite.inMemoryPersistence` Disable persisting any data to disk. If the Azurite process is terminated, all data is lost.
+- `azurite.extentMemoryLimit` When using in-memory persistence, limit the total size of extents (blob and queue content) to a specific number of megabytes. This does not limit blob, queue, or table metadata. Defaults to 50% of total memory.
 
 ### [DockerHub](https://hub.docker.com/_/microsoft-azure-storage-azurite)
 
@@ -430,6 +434,52 @@ Optional. When using FQDN instead of IP in request Uri host, by default Azurite
 --disableProductStyleUrl
 ```
 
+### Use in-memory storage
+
+Optional. Disable persisting any data to disk and only store data in-memory. If the Azurite process is terminated, all
+data is lost. By default, LokiJS persists blob and queue metadata to disk and content to extent files. Table storage
+persists all data to disk. This behavior can be disabled using this option. This setting is rejected when the SQL based
+metadata implementation is enabled (via `AZURITE_DB`). This setting is rejected when the `--location` option is
+specified.
+
+```cmd
+--inMemoryPersistence
+```
+
+By default, the in-memory extent store (for blob and queue content) is limited to 50% of the total memory on the host
+machine. This is evaluated to using [`os.totalmem()`](https://nodejs.org/api/os.html#ostotalmem). This limit can be
+overridden using the `--extentMemoryLimit <megabytes>` option. There is no restriction on the value specified for this
+option but virtual memory may be used if the limit exceeds the amount of available physical memory as provided by the
+operating system. A high limit may eventually lead to out of memory errors or reduced performance.
+
+As blob or queue content (i.e. bytes in the in-memory extent store) is deleted, the memory is not freed immediately.
+Similar to the default file-system based extent store, both the blob and queue service have an extent garbage collection
+(GC) process. This process is in addition to the standard Node.js runtime GC. The extent GC periodically detects unused
+extents and deletes them from the extent store. This happens on a regular time period rather than immediately after
+the blob or queue REST API operation that caused some content to be deleted. This means that process memory consumed by
+the deleted blob or queue content will only be released after both the extent GC and the runtime GC have run. The extent
+GC will remove the reference to the in-memory byte storage and the runtime GC will free the unreferenced memory some
+time after that. The blob extent GC runs every 10 minutes and the queue extent GC runs every 1 minute.
+
+The queue and blob extent storage count towards the same limit. The `--extentMemoryLimit` setting is rejected when
+`--inMemoryPersistence` is not specified. LokiJS storage (blob and queue metadata and table data) does
+not contribute to this limit and is unbounded which is the same as without the `--inMemoryPersistence` option.
+
+```cmd
+--extentMemoryLimit <megabytes>
+```
+
+This option is rejected when `--inMemoryPersistence` is not specified.
+
+When the limit is reached, write operations to the blob or queue endpoints which carry content will fail with an `HTTP
+409` status code, a custom storage error code of `MemoryExtentStoreAtSizeLimit`, and a helpful error message.
+Well-behaved storage SDKs and tools will not a retry on this failure and will return a related error message. If this
+error is met, consider deleting some in-memory content (blobs or queues), raising the limit, or restarting the Azurite
+server thus resetting the storage completely.
+
+Note that if many hundreds of megabytes of content (queue message or blob content) are stored in-memory, it can take
+noticeably longer than usual for the process to terminate since all the consumed memory needs to be released.
+
 ### Command Line Options Differences between Azurite V2
 
 Azurite V3 supports SharedKey, Account Shared Access Signature (SAS), Service SAS, OAuth, and Public Container Access authentications, you can use any Azure Storage SDKs or tools like Storage Explorer to connect Azurite V3 with any authentication strategy.

azure-pipelines.yml

@@ -38,6 +38,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestubuntu22_04
     displayName: Blob Test Linux Ubuntu 22.04
     pool:
@@ -65,6 +71,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestwin
     displayName: Blob Test Windows
     pool:
@@ -92,6 +104,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestmac
     displayName: Blob Test Mac
     pool:
@@ -119,6 +137,12 @@ jobs:
         displayName: "npm run test:blob"
         env: {}
 
+      - script: |
+          npm run test:blob:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:blob:in-memory"
+        env: {}
+
   - job: blobtestmysql
     displayName: Blob Test Mysql
     pool:
@@ -181,6 +205,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: queuetestwin
     displayName: Queue Test Windows
     pool:
@@ -208,6 +238,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: queuetestmac
     displayName: Queue Test Mac
     pool:
@@ -235,6 +271,12 @@ jobs:
         displayName: "npm run test:queue"
         env: {}
 
+      - script: |
+          npm run test:queue:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:queue:in-memory"
+        env: {}
+
   - job: tabletestlinux
     displayName: Table Test Linux
     pool:
@@ -263,6 +305,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: tabletestwin
     displayName: Table Test Windows
     pool:
@@ -291,6 +339,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: tabletestmac
     displayName: Table Test Mac
     pool:
@@ -319,6 +373,12 @@ jobs:
         displayName: "npm run test:table"
         env: {}
 
+      - script: |
+          npm run test:table:in-memory
+        workingDirectory: "./"
+        displayName: "npm run test:table:in-memory"
+        env: {}
+
   - job: azuritenodejslinux
     displayName: Azurite Linux
     pool: