Skip to content

Added Bulk Job Scripts for Data Analysis#6

Closed
bradtaylorsf wants to merge 72 commits into
feature/beta-v1from
bt/bulkjobs
Closed

Added Bulk Job Scripts for Data Analysis#6
bradtaylorsf wants to merge 72 commits into
feature/beta-v1from
bt/bulkjobs

Conversation

@bradtaylorsf
Copy link
Copy Markdown
Collaborator

Added a new Contentful model called Fine Tuned Question which is a name value pair.
This includes scripts for bulk workflows running with different settings for each job

maxtechera and others added 30 commits March 28, 2024 16:46
Add support for Airtable text field content

answerai brand

added contentful

removed publisher from this branch

removed textField from the node to fix errors

Update docker-compose to build the repo

Update image

added youtube document loader

added updated contentful loader

Add chatflow API key and sync with Sidekick on create/update

Update .gitignore and constant.js

Update API key functionality and add chatflow domain and API key to request payload

added hidden nav and manu when in iframe

Remove command from compose

optimize dockkerfile

Make dark theme the default

Add DOMAIN env

Add API_KEY env

Add DOMAIN env

Add auth0 integration

Update dockerfile

Add env passthrough for auth

Add env passthrough for auth

Add environment variables for authentication in production

Add production environment variables

Update compose env

Update authentication logging and chatflow domain

Update Dockerfile and env.sh, fix sed command in env.sh, and modify index.js to handle undefined organization ID

Refactor Dockerfile and env.sh scripts, and update App.js and index.js files

Add embed & embed-react, enable credentials for API comms

Improvements

Add SST

Fix authorization issue in API endpoints

Update .gitignore file to ignore .pem files and .env.* files

added option for string or document fo routput and adjusted config

Add lock files to git

Update Docker files

Add CDN and remove quotes from chat prompt so JSON is not malformed

updated to account for queryoveride
@maxtechera maxtechera force-pushed the feature/beta-v1 branch 2 times, most recently from 1c1c23e to 5fca7f0 Compare June 4, 2024 15:25
@bradtaylorsf bradtaylorsf deleted the bt/bulkjobs branch June 26, 2024 23:56
maxtechera pushed a commit that referenced this pull request Aug 30, 2024
* added meilisearch retriever and credentials.ts

* added semantic ratio

* removed a TODO implementatio

* meilisearch component implemented with searching and upsert functionality (#3)

meilisearch retriever component created , searching for an existing index and upserting a new or existing index has been implemented , component utilizes langchain and meilisearch vector search

Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/3
Reviewed-by: mohamed1999akram <mohamed1999akram@gmail.com>

* added CI/CD for ntg branch, added proper dockerfile for flowise-ntg (#4)

Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/4
Reviewed-by: mohammad <mohammad@noreply.git.beyond.cc>

* modified os version , removed linting errors , removed cypress github actions (#5)

added --no-lock-file flag to pass CICD , made the runner run on debian and not ubuntu , removed code that caused warnings to pass linting

Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/5
Reviewed-by: omaryassery <omarryassser@gmail.com>

* removed unnecessary QEMU install action (#6)

Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/6
Reviewed-by: omaryassery <omarryassser@gmail.com>

* removed cypress installation and linting from dockerfile (#7)

Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/7
Reviewed-by: isameh <isameh@ntgclarity.com>

* dockerfile-ntg-modification (#9)

dockerfile-ntg modified to copy all working directory before calling pnpm install
Reviewed-on: https://git.beyond.cc/ntg/flowise/pulls/9
Reviewed-by: isameh <isameh@ntgclarity.com>

* resolved comments, reverted CI/CD

* add test docker build yml back

* moved meilisearch to vector store folder

* Update Meilisearch.ts

---------

Co-authored-by: Henry <hzj94@hotmail.com>
Co-authored-by: Henry Heng <henryheng@flowiseai.com>
maxtechera added a commit that referenced this pull request Aug 5, 2025
- SSRF #29: FlowiseStream.ts - Added URL validation before fetch
- SSRF #28: utils.ts xmlScrape - Added isSecureURL validation
- SSRF #27: utils.ts crawl - Added isSecureURL validation
- Type Confusion #6: getChatMessage.ts - Added Array.isArray check

Security improvements:
- Created centralized URL validation in validator.ts
- Blocks localhost, private IPs, cloud metadata endpoints
- Type-safe enum validation to prevent parameter tampering
- Minimal, surgical fixes without breaking functionality

Fixes GitHub code scanning alerts #29, #28, #27, #6
maxtechera added a commit that referenced this pull request Aug 5, 2025
- SSRF #29: FlowiseStream.ts - Added URL validation before fetch
- SSRF #28: utils.ts xmlScrape - Added isSecureURL validation
- SSRF #27: utils.ts crawl - Added isSecureURL validation
- Type Confusion #6: getChatMessage.ts - Added Array.isArray check

Security improvements:
- Created centralized URL validation in validator.ts
- Blocks localhost, private IPs, cloud metadata endpoints
- Type-safe enum validation to prevent parameter tampering
- Minimal, surgical fixes without breaking functionality

Fixes GitHub code scanning alerts #29, #28, #27, #6
ct3685 added a commit that referenced this pull request Aug 6, 2025
…ul (#456)

* fix: resolve 4 critical security vulnerabilities (CodeQL)

- SSRF #29: FlowiseStream.ts - Added URL validation before fetch
- SSRF #28: utils.ts xmlScrape - Added isSecureURL validation
- SSRF #27: utils.ts crawl - Added isSecureURL validation
- Type Confusion #6: getChatMessage.ts - Added Array.isArray check

Security improvements:
- Created centralized URL validation in validator.ts
- Blocks localhost, private IPs, cloud metadata endpoints
- Type-safe enum validation to prevent parameter tampering
- Minimal, surgical fixes without breaking functionality

Fixes GitHub code scanning alerts #29, #28, #27, #6

* Remove sha from packageManager version to prevent conflicts

* Fix import for validateEnumArray

* chore: add SonarCloud analysis workflow for CI/CD integration

* chore: update embed submodule to latest commit for improved stability

* chore: update embed submodule to track ct/staging-pnpm-recursive-updates branch for temporary build check

* chore: update dependencies and submodules for improved stability and security

- Update embed submodule to track ct/staging-pnpm-recursive-updates branch
- Upgrade various dependencies in package.json files across multiple packages to their latest versions for better performance and security
- Ensure consistency in package versions and resolve potential vulnerabilities

* chore: update dependencies in package.json and pnpm-lock.yaml for improved compatibility and security

- Upgrade cypress to version 13.17.0 for enhanced testing capabilities
- Update lint-staged to version 15.2.0 for better performance
- Modify dependency versions in pnpm-lock.yaml to use minimum required versions for better stability
- Remove deprecated request package from packages-answers/utils/package.json to streamline dependencies

* chore: update dependencies in package.json and pnpm-lock.yaml for improved compatibility and performance

- Add postcss@8.4.31, tsup@8.3.5, and jsforce@3.10.0 to package.json for enhanced functionality
- Remove react-scripts from packages/ui/package.json to streamline dependencies
- Upgrade tsup in packages-answers/ui/package.json to version 8.3.5 for better build performance
- Modify pnpm-lock.yaml to reflect updated dependency versions for consistency and stability

* chore: enhance BWS Secure integration and update dependencies for improved functionality

- Refactor BWS Secure scripts to utilize execBwsCommandWithRetrySync for better error handling and retry logic
- Update README.md for clearer instructions on creating access tokens
- Upgrade dotenv, dotenv-cli, yargs, and axios in package.json for compatibility and performance improvements
- Add bws-retry-utils.js for managing rate limits and retry logic in BWS commands
- Ensure consistent logging and error handling across BWS Secure scripts

* chore: update TypeScript configuration to include ES2022 library for enhanced compatibility

* chore: update Dockerfile to improve build stage clarity

- Change 'as' to 'AS' for consistency in stage definitions
- Ensure clarity in multi-stage build process by standardizing syntax

* chore: update embed submodule to track a-main branch for merge

- Change submodule branch from ct/staging-pnpm-recursive-updates to a-main
- Update subproject commit reference for improved stability

* chore:

- Add synchronous sleep function in bws-retry-utils.js for improved rate limit handling

* chore: update SonarCloud workflow for improved checkout process

- Upgrade actions/checkout from v3 to v4 for enhanced functionality
- Set fetch-depth to 0 for complete history retrieval during checkout

* chore: remove SonarCloud workflow file

- Delete the SonarCloud analysis workflow file, running CI analysis with Automatic Analysis.

* chore: standardize Dockerfile stage definitions

- Change 'AS' to 'as' for consistency in stage definitions
- Improve clarity in multi-stage build process by standardizing syntax

* chore: update pnpm-lock.yaml for dependency consistency and version alignment

* fix: resolve CLI command discovery by removing path-to-regexp override

- Remove 'path-to-regexp: "0.1.12"' from pnpm.overrides in package.json
- This was forcing @oclif/core@4.0.7 to use old path-to-regexp version without .match() method
- Fixes 'pathRegexp.match is not a function' error that prevented CLI command registration
- CLI now properly discovers start/worker commands when environment variables are provided
- Revert multer-cloud-storage back to ^4.1.0 (was incorrectly blamed for the issue)
- Deployment should now work as CLI commands are properly discoverable

---------

Co-authored-by: Max Techera <maxi.techerag@gmail.com>
maxtechera added a commit that referenced this pull request Oct 21, 2025
Critical & High Priority Fixes:
- Update documentation to reflect auto-tagging implementation (Issue #1)
- Fix skipped traces double-counting bug with else-if pattern (Issue #3)

Medium Priority Fixes:
- Reduce self-healing log spam with aggregated logging (Issue #4)
- Remove unnecessary allTraces/allCreditsData memory accumulation (Issue #2)

Low Priority Fixes:
- Fix potential race condition with Set-based tag deduplication (Issue #5)
- Remove deprecated fetchUsageData method - 119 lines deleted (Issue #6)

Code Quality Improvements:
- Add getTraceTags() type safety helper method
- Replace magic number with FUTURE_TIMESTAMP_BUFFER_SECONDS constant
- Improve code maintainability and readability

Net Impact: -103 lines, all PR #615 review issues resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants