Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strategy modifcations #57

Merged
merged 21 commits into from
Mar 12, 2024
Merged

Strategy modifcations #57

merged 21 commits into from
Mar 12, 2024

Conversation

Faouzijedidi1
Copy link
Contributor

@Faouzijedidi1 Faouzijedidi1 commented Mar 11, 2024

User description

  • Fix CORS issue with the gateway
  • Add a fix to the custom logger file saving by switching to winston logging
  • Merge server/README to root README
  • Add root directory package.json

Strategy improvement requested as improvements by CHUNKEE

  • Ceiling and floor graceful handling, no shutdown skip orders but don't stop strategy
  • Cancel all orders once strategy shutdown, // Cancel method need to modified to handle multiple users.
  • Improved error handling with more retries as exchanges tend to have issues.

Type

enhancement, bug_fix


Description

  • Added SSL support for PostgreSQL connections, enhancing security.
  • Switched to winston for logging, improving log management.
  • Allowed all origins in WebSocket CORS settings for development ease.
  • Introduced ceilingPrice and floorPrice in strategy DTO to manage order placement boundaries.
  • Enhanced strategy service to cancel all orders on shutdown and retry on errors, improving reliability.
  • Merged server README into root README and updated to use yarn, enhancing project documentation.
  • Added a root directory package.json to simplify project setup and management.

Changes walkthrough

Relevant files
Enhancement
app.module.ts
Add SSL Support for PostgreSQL Connection                               

server/src/app.module.ts

  • Added SSL support for PostgreSQL connection based on environment
    variable POSTGRES_SSL.
  • +1/-0     
    logger.service.ts
    Switch Logger to Use Winston                                                         

    server/src/modules/logger/logger.service.ts

  • Switched logging mechanism to use winston instead of native file
    system operations.
  • Configured winston to log both to console and file with different
    levels.
  • +29/-45 
    marketdata.gateway.ts
    Modify WebSocket CORS Settings                                                     

    server/src/modules/marketdata/marketdata.gateway.ts

    • Modified WebSocket CORS settings to allow all origins.
    +4/-2     
    strategy.dto.ts
    Add Ceiling and Floor Price Handling to Strategy DTO         

    server/src/modules/strategy/strategy.dto.ts

  • Added ceilingPrice and floorPrice properties to strategy DTO for
    handling order placement boundaries.
  • +4/-6     
    strategy.service.ts
    Improve Strategy Shutdown and Error Handling                         

    server/src/modules/strategy/strategy.service.ts

  • Implemented cancellation of all orders upon strategy shutdown.
  • Added error handling with retries for order book watching and strategy
    execution.
  • Modified market making strategy to skip orders outside of specified
    ceiling and floor prices instead of shutting down.
  • +116/-66
    package.json
    Add Root Directory Package.json                                                   

    package.json

  • Added root directory package.json for managing both frontend and
    server with scripts for convenience.
  • +13/-0   
    Documentation
    README.md
    Merge Server README to Root and Update Instructions           

    README.md

  • Merged server README content into root README.
  • Updated instructions to use yarn instead of npm.
  • +41/-2   

    PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    Copy link

    vercel bot commented Mar 11, 2024

    The latest updates on your projects. Learn more about Vercel for Git ↗︎

    Name Status Preview Comments Updated (UTC)
    mr-market ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 12, 2024 2:22pm

    @github-actions github-actions bot added enhancement New feature or request bug_fix labels Mar 11, 2024
    Copy link

    PR Description updated to latest commit (adc2259)

    Copy link

    github-actions bot commented Mar 11, 2024

    PR Review

    (Review updated until commit 723e319)

    ⏱️ Estimated effort to review [1-5]

    4, due to the comprehensive changes across multiple files, including significant logic adjustments in strategy handling, logging, and server configuration. The introduction of new features and the switch to a different logging mechanism increase the complexity of the review. Additionally, the modifications in error handling and strategy shutdown procedures require careful consideration to ensure reliability and robustness.

    🧪 Relevant tests

    No

    🔍 Possible issues

    Possible Bug: In server/src/modules/strategy/strategy.service.ts, the retry mechanism in error handling could potentially lead to infinite retries if the error persists. This could cause the system to hang or consume excessive resources.

    Performance Concern: The use of synchronous await inside loops, for example in cancelAllStrategyOrders and manageMarketMakingOrdersWithLayers, could lead to performance bottlenecks. Asynchronous batch processing or parallel execution strategies might be more efficient.

    🔒 Security concerns

    No

    Code feedback:
    relevant fileserver/src/modules/logger/logger.service.ts
    suggestion      

    Consider adding a file rotation mechanism or log cleanup strategy to prevent the log files from growing indefinitely, especially for long-running applications. This can be achieved by integrating winston-daily-rotate-file transport or implementing a custom cleanup mechanism. [important]

    relevant linenew winston.transports.File({

    relevant fileserver/src/modules/marketdata/marketdata.gateway.ts
    suggestion      

    For production readiness, ensure that the CORS configuration is updated to restrict origins to known and trusted sources instead of allowing all origins. This is crucial for preventing unwanted cross-origin requests. [important]

    relevant lineorigin: '*', // Allow all origins, Temporary to be changed and restricted.

    relevant fileserver/src/modules/strategy/strategy.service.ts
    suggestion      

    Implement a maximum retry limit for error handling in methods like watchSymbols and manageMarketMakingOrdersWithLayers to prevent infinite loops in case of persistent errors. This could be done by adding a retry counter and a condition to break out of the loop after reaching the limit. [important]

    relevant lineawait new Promise((resolve) => setTimeout(resolve, 2000)); // Wait for 2 seconds before retrying or moving on

    relevant fileserver/src/app.module.ts
    suggestion      

    Validate the POSTGRES_SSL environment variable to ensure it contains a valid boolean string ('true' or 'false'). Incorrect values could lead to unexpected behavior. Consider adding a utility function for boolean environment variables parsing. [medium]

    relevant linessl: process.env.POSTGRES_SSL === 'true',


    ✨ Review tool usage guide:

    Overview:
    The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
    When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

    /review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...
    

    With a configuration file, use the following template:

    [pr_reviewer]
    some_config1=...
    some_config2=...
    
    Utilizing extra instructions

    The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project.

    Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize.

    Examples for extra instructions:

    [pr_reviewer] # /review #
    extra_instructions="""
    In the 'possible issues' section, emphasize the following:
    - Does the code logic cover relevant edge cases?
    - Is the code logic clear and easy to understand?
    - Is the code logic efficient?
    ...
    """
    

    Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

    How to enable\disable automation
    • When you first install PR-Agent app, the default mode for the review tool is:
    pr_commands = ["/review", ...]
    

    meaning the review tool will run automatically on every PR, with the default configuration.
    Edit this field to enable/disable the tool, or to change the used configurations

    Auto-labels

    The review tool can auto-generate two specific types of labels for a PR:

    • a possible security issue label, that detects possible security issues (enable_review_labels_security flag)
    • a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)
    Extra sub-tools

    The review tool provides a collection of possible feedbacks about a PR.
    It is recommended to review the possible options, and choose the ones relevant for your use case.
    Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example:
    require_score_review, require_soc2_ticket, and more.

    Auto-approve PRs

    By invoking:

    /review auto_approve
    

    The tool will automatically approve the PR, and add a comment with the approval.

    To ensure safety, the auto-approval feature is disabled by default. To enable auto-approval, you need to actively set in a pre-defined configuration file the following:

    [pr_reviewer]
    enable_auto_approval = true
    

    (this specific flag cannot be set with a command line argument, only in the configuration file, committed to the repository)

    You can also enable auto-approval only if the PR meets certain requirements, such as that the estimated_review_effort is equal or below a certain threshold, by adjusting the flag:

    [pr_reviewer]
    maximal_review_effort = 5
    
    More PR-Agent commands

    To invoke the PR-Agent, add a comment using one of the following commands:

    • /review: Request a review of your Pull Request.
    • /describe: Update the PR title and description based on the contents of the PR.
    • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
    • /ask <QUESTION>: Ask a question about the PR.
    • /update_changelog: Update the changelog based on the PR's contents.
    • /add_docs 💎: Generate docstring for new components introduced in the PR.
    • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
    • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

    See the tools guide for more details.
    To list the possible configuration parameters, add a /config comment.

    See the review usage page for a comprehensive guide on using this tool.

    Copy link

    github-actions bot commented Mar 11, 2024

    PR Code Suggestions

    CategorySuggestions                                                                                                                                                       
    Enhancement
    Improve security and reliability by using a configuration service for environment variables.

    Consider using a more secure method for configuring SSL in production environments. Using
    environment variables directly can be prone to errors and might not be secure. A better
    approach would be to use a configuration service or module that validates and sanitizes
    all environment variables before they are used in the application.

    server/src/app.module.ts [48]

    -ssl: process.env.POSTGRES_SSL === 'true',
    +ssl: configService.get('POSTGRES_SSL') === 'true',
     
    Implement log file rotation to manage disk space and log file sizes.

    Implement a mechanism to handle log file rotation to prevent log files from growing
    indefinitely. This can be achieved by using the winston-daily-rotate-file transport
    instead of the basic File transport. This will help in managing disk space more
    effectively and ensure that the log files are easier to manage and review.

    server/src/modules/logger/logger.service.ts [28-34]

    -new winston.transports.File({
    -  filename: path.join(logsDir, 'error.log'),
    +new winston.transports.DailyRotateFile({
    +  filename: path.join(logsDir, '%DATE%-error.log'),
    +  datePattern: 'YYYY-MM-DD',
       level: 'error',
     }),
    -new winston.transports.File({
    -  filename: path.join(logsDir, 'combined.log'),
    +new winston.transports.DailyRotateFile({
    +  filename: path.join(logsDir, '%DATE%-combined.log'),
    +  datePattern: 'YYYY-MM-DD',
     }),
     
    Implement an exponential backoff strategy for retries to improve performance and reliability.

    Using a fixed delay for retry mechanisms (setTimeout(resolve, 2000)) can lead to
    suboptimal performance under varying network conditions or load. Implementing an
    exponential backoff strategy for retries can be more effective, as it adapts to the
    situation by gradually increasing the delay between retries, reducing the load on the
    server and increasing the chance of recovery in case of temporary issues.

    server/src/modules/strategy/strategy.service.ts [177]

    -await new Promise((resolve) => setTimeout(resolve, 2000)); // Wait for 2 seconds before retrying or moving on
    +await this.exponentialBackoffRetry(exchange, symbol);
     
    Add logic to handle scenarios where the price source is outside the specified ceiling and floor prices.

    The current implementation does not handle the scenario where the price source is outside
    the specified ceiling and floor prices. It's crucial to add logic to handle this scenario
    to prevent placing orders that do not align with the strategy's constraints. This could
    involve logging a warning or error and skipping the order placement for that cycle.

    server/src/modules/strategy/strategy.service.ts [257]

    -// Fetch the current market price based on the specified price source type
    +if (priceSource > ceilingPrice || priceSource < floorPrice) {
    +  this.logger.warn(`Price source ${priceSource} is outside the specified range (Floor: ${floorPrice}, Ceiling: ${ceilingPrice}). Skipping order placement.`);
    +  return;
    +}
     
    Security
    Enhance security by restricting CORS to specific origins.

    Restricting CORS to allow all origins ('*') can expose the service to unnecessary security
    risks. It's recommended to specify a list of allowed origins or to configure CORS
    dynamically based on the request. This can prevent unwanted cross-origin requests and
    enhance the security of the WebSocket service.

    server/src/modules/marketdata/marketdata.gateway.ts [24-25]

     cors: {
    -  origin: '*', // Allow all origins, Temporary to be changed and restricted.
    +  origin: ['https://example.com', 'https://anotherdomain.com'], // Specify allowed origins
     },
     

    ✨ Improve tool usage guide:

    Overview:
    The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.
    When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

    /improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...
    

    With a configuration file, use the following template:

    [pr_code_suggestions]
    some_config1=...
    some_config2=...
    
    Enabling\disabling automation

    When you first install the app, the default mode for the improve tool is:

    pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...]
    

    meaning the improve tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.

    Utilizing extra instructions

    Extra instructions are very important for the improve tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project.

    Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on.

    Examples for extra instructions:

    [pr_code_suggestions] # /improve #
    extra_instructions="""
    Emphasize the following aspects:
    - Does the code logic cover relevant edge cases?
    - Is the code logic clear and easy to understand?
    - Is the code logic efficient?
    ...
    """
    

    Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

    A note on code suggestions quality
    • While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
    • Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base.
    • Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the custom suggestions 💎 tool
    • With large PRs, best quality will be obtained by using 'improve --extended' mode.
    More PR-Agent commands

    To invoke the PR-Agent, add a comment using one of the following commands:

    • /review: Request a review of your Pull Request.
    • /describe: Update the PR title and description based on the contents of the PR.
    • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
    • /ask <QUESTION>: Ask a question about the PR.
    • /update_changelog: Update the changelog based on the PR's contents.
    • /add_docs 💎: Generate docstring for new components introduced in the PR.
    • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
    • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

    See the tools guide for more details.
    To list the possible configuration parameters, add a /config comment.

    See the improve usage page for a more comprehensive guide on using this tool.

    Copy link

    PR Description updated to latest commit (723e319)

    Copy link

    Persistent review updated to latest commit 723e319

    Copy link

    github-actions bot commented Mar 12, 2024

    PR Code Suggestions

    CategorySuggestions                                                                                                                                                       
    Enhancement
    Set a default value for ssl connection option based on the environment variable.

    Consider setting ssl to false by default if the POSTGRES_SSL environment variable is not
    set. This ensures that your application can still connect to the database in environments
    where SSL is not required or not supported, without requiring explicit configuration.

    server/src/app.module.ts [48]

    -ssl: process.env.POSTGRES_SSL === 'true',
    +ssl: process.env.POSTGRES_SSL ? process.env.POSTGRES_SSL === 'true' : false,
     
    Ensure the logs directory exists before writing logs.

    Implement a mechanism to ensure that the logs directory exists before attempting to write
    logs to it. This can prevent runtime errors when the specified directory does not exist.

    server/src/modules/logger/logger.service.ts [11-13]

     const logsDir = process.env.IS_DEV
         ? path.join(__dirname, '..', '..', 'logs')
         : path.join(__dirname, '..', 'logs');
    +if (!fs.existsSync(logsDir)) {
    +    fs.mkdirSync(logsDir, { recursive: true });
    +}
     
    Implement exponential backoff for retrying operations after an error.

    Instead of using a fixed delay for retrying operations after an error, consider
    implementing an exponential backoff strategy. This approach can help to reduce the load on
    the server and improve the chances of recovery in case of temporary issues.

    server/src/modules/strategy/strategy.service.ts [177]

    -await new Promise((resolve) => setTimeout(resolve, 2000)); // Wait for 2 seconds before retrying or moving on
    +await new Promise((resolve) => setTimeout(resolve, Math.pow(2, retryAttempt) * 1000)); // Exponential backoff
     
    Security
    Restrict CORS origins to specific domains for enhanced security.

    Replace the wildcard CORS origin with specific origins to enhance security. Allowing all
    origins ('*') can expose your service to cross-site request forgery (CSRF) attacks.

    server/src/modules/marketdata/marketdata.gateway.ts [24]

    -origin: '*', // Allow all origins, Temporary to be changed and restricted.
    +origin: ['http://example.com', 'https://anotherdomain.com'], // Specify allowed origins
     
    Maintainability
    Use descriptive variable names for better code readability.

    Use a more descriptive variable name than priceSource for the variable that holds the
    price used to calculate buy and sell prices. This will improve code readability and
    maintainability.

    server/src/modules/strategy/strategy.service.ts [258-262]

    -const priceSource = await this.getPriceSource(
    +const referencePrice = await this.getPriceSource(
     

    ✨ Improve tool usage guide:

    Overview:
    The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.
    When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

    /improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...
    

    With a configuration file, use the following template:

    [pr_code_suggestions]
    some_config1=...
    some_config2=...
    
    Enabling\disabling automation

    When you first install the app, the default mode for the improve tool is:

    pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...]
    

    meaning the improve tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.

    Utilizing extra instructions

    Extra instructions are very important for the improve tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project.

    Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on.

    Examples for extra instructions:

    [pr_code_suggestions] # /improve #
    extra_instructions="""
    Emphasize the following aspects:
    - Does the code logic cover relevant edge cases?
    - Is the code logic clear and easy to understand?
    - Is the code logic efficient?
    ...
    """
    

    Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

    A note on code suggestions quality
    • While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
    • Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base.
    • Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the custom suggestions 💎 tool
    • With large PRs, best quality will be obtained by using 'improve --extended' mode.
    More PR-Agent commands

    To invoke the PR-Agent, add a comment using one of the following commands:

    • /review: Request a review of your Pull Request.
    • /describe: Update the PR title and description based on the contents of the PR.
    • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
    • /ask <QUESTION>: Ask a question about the PR.
    • /update_changelog: Update the changelog based on the PR's contents.
    • /add_docs 💎: Generate docstring for new components introduced in the PR.
    • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
    • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

    See the tools guide for more details.
    To list the possible configuration parameters, add a /config comment.

    See the improve usage page for a more comprehensive guide on using this tool.

    @posix4e posix4e merged commit 9f57d6d into main Mar 12, 2024
    8 checks passed
    @posix4e posix4e deleted the strategy-modifcations branch March 12, 2024 14:30
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    3 participants