Skip to content

Feat: add Google Cloud Storage Connector #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

Feat: add Google Cloud Storage Connector #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b4d1a3d
feat: add Google Cloud Storage connector
SyedZawwarAhmed Sep 1, 2025
73093d9
docs: add Google Cloud Storage connector documentation
SyedZawwarAhmed Sep 1, 2025
8395156
refactor: simplify GCS Storage by removing lifecycle helper and using…
SyedZawwarAhmed Sep 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ SmythOS provides a complete **Operating System for Agentic AI**. Just as traditi

### Unified Resource Abstraction

SmythOS provides a **unified interface for all resources**, ensuring consistency and simplicity across your entire AI platform. Whether you're storing a file locally, on S3, or any other storage provider, you don't need to worry about the underlying implementation details. SmythOS offers a powerful abstraction layer where all providers expose the same functions and APIs.
SmythOS provides a **unified interface for all resources**, ensuring consistency and simplicity across your entire AI platform. Whether you're storing a file locally, on S3, GCS, or any other storage provider, you don't need to worry about the underlying implementation details. SmythOS offers a powerful abstraction layer where all providers expose the same functions and APIs.

This principle applies to **all services** - not just storage. Whether you're working with VectorDBs, cache (Redis, RAM), LLMs (OpenAI, Anthropic), or any other resource, the interface remains consistent across providers.

Expand Down
20 changes: 20 additions & 0 deletions examples/06-Storage-no-agent/03-GCS.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { Storage } from '@smythos/sdk';

async function main() {
const gcsStorage = Storage.GCS({
projectId: process.env.GCP_PROJECT_ID,
clientEmail: process.env.GCP_CLIENT_EMAIL,
privateKey: process.env.GCP_PRIVATE_KEY,
bucket: process.env.GCP_BUCKET_NAME,
Comment on lines +5 to +8
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Env private key likely breaks without newline restoration

Most env-injected SA keys require replace(/\\n/g, '\n'). Without it, auth fails.

Apply:

-        privateKey: process.env.GCP_PRIVATE_KEY,
+        privateKey: process.env.GCP_PRIVATE_KEY?.replace(/\\n/g, '\n'),
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
projectId: process.env.GCP_PROJECT_ID,
clientEmail: process.env.GCP_CLIENT_EMAIL,
privateKey: process.env.GCP_PRIVATE_KEY,
bucket: process.env.GCP_BUCKET_NAME,
projectId: process.env.GCP_PROJECT_ID,
clientEmail: process.env.GCP_CLIENT_EMAIL,
privateKey: process.env.GCP_PRIVATE_KEY?.replace(/\\n/g, '\n'),
bucket: process.env.GCP_BUCKET_NAME,
🤖 Prompt for AI Agents 
In examples/06-Storage-no-agent/03-GCS.ts around lines 5 to 8 the service
account privateKey pulled from process.env.GCP_PRIVATE_KEY will fail auth if
newline characters are escaped; update the assignment to restore real newlines
by replacing escaped sequences (e.g. call replace(/\\n/g, '\n') on
process.env.GCP_PRIVATE_KEY) before passing it to the client constructor,
ensuring you handle undefined safely (e.g. fallback or throw a clear error if
the env var is missing).

});

await gcsStorage.write('test.txt', 'Hello, world!');

const data = await gcsStorage.read('test.txt');

const dataAsString = data.toString();

console.log(dataAsString);
}

main();
47 changes: 47 additions & 0 deletions packages/core/docs/connectors/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,50 @@ SRE.init({
- Store credentials securely using environment variables or AWS Secrets Manager
- Configure appropriate bucket policies and CORS settings
- Enable encryption at rest and in transit for sensitive data

---

### GCS

**Role**: Google Cloud Storage connector
**Summary**: Provides scalable cloud storage using Google Cloud Storage, suitable for production deployments requiring high availability and durability on Google Cloud Platform.

| Setting | Type | Required | Default | Description |
| ------------- | ------ | -------- | ------- | ----------------------------------------------- |
| `projectId` | string | Yes | - | Google Cloud Project ID where the bucket is located |
| `clientEmail` | string | Yes | - | Service account email address |
| `privateKey` | string | Yes | - | Service account private key |
| `bucket` | string | Yes | - | GCS bucket name for storing files |

**Example Configuration:**

```typescript
import { SRE } from '@smythos/sre';

SRE.init({
Storage: {
Connector: 'GCS',
Settings: {
projectId: 'my-project-id',
clientEmail: process.env.GCP_CLIENT_EMAIL,
privateKey: process.env.GCP_PRIVATE_KEY,
bucket: 'my-app-storage',
},
},
});
```
Comment on lines +101 to +115
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Private key newline gotcha (common failure in env vars)

Service account keys stored in env vars often need newline restoration. Add a tip:

-            privateKey: process.env.GCP_PRIVATE_KEY,
+            // If stored as single-line env var, restore newlines:
+            privateKey: process.env.GCP_PRIVATE_KEY?.replace(/\\n/g, '\n'),

I can update the example + add a warning callout.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```typescript
import { SRE } from '@smythos/sre';
SRE.init({
Storage: {
Connector: 'GCS',
Settings: {
projectId: 'my-project-id',
clientEmail: process.env.GCP_CLIENT_EMAIL,
privateKey: process.env.GCP_PRIVATE_KEY,
bucket: 'my-app-storage',
},
},
});
```
🤖 Prompt for AI Agents 
In packages/core/docs/connectors/storage.md around lines 101 to 115, the example
uses process.env.GCP_PRIVATE_KEY directly which commonly fails because
environment variables lose literal newlines; update the docs to include a short
tip and example showing restoration of newlines (e.g., replace escaped "\\n"
with actual "\n" before use) and add a brief warning callout about storing
service account keys in env vars and needing to restore newlines so the private
key is parsed correctly by GCP clients.


**Use Cases:**

- Production environments requiring scalability on Google Cloud Platform
- Multi-region deployments within GCP infrastructure
- Applications with high availability requirements
- Integration with Google Cloud ecosystem
- Large-scale data storage and processing

**Security Notes:**

- Use service accounts with minimal required permissions
- Store credentials securely using environment variables or Google Secret Manager
- Configure appropriate bucket policies and IAM settings
- Enable encryption at rest and in transit for sensitive data
1 change: 1 addition & 0 deletions packages/core/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
"@aws-sdk/client-s3": "^3.826.0",
"@aws-sdk/client-secrets-manager": "^3.826.0",
"@faker-js/faker": "^9.8.0",
"@google-cloud/storage": "^7.17.0",
"@google-cloud/vertexai": "^1.7.0",
"@google/genai": "^1.10.0",
"@google/generative-ai": "^0.14.1",
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ export * from './subsystems/IO/NKV.service/connectors/NKVRAM.class';
export * from './subsystems/IO/NKV.service/connectors/NKVRedis.class';
export * from './subsystems/IO/Router.service/connectors/ExpressRouter.class';
export * from './subsystems/IO/Router.service/connectors/NullRouter.class';
export * from './subsystems/IO/Storage.service/connectors/GCSStorage.class';
export * from './subsystems/IO/Storage.service/connectors/LocalStorage.class';
export * from './subsystems/IO/Storage.service/connectors/S3Storage.class';
export * from './subsystems/IO/VectorDB.service/connectors/MilvusVectorDB.class';
Expand Down
Loading