Feature/smart ohlcv conversion #24
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement API client for PyneSys compiler service with full functionality including: - Authentication and token validation - Pine Script compilation with error handling - Configuration management - File utilities for compilation results - CLI commands for API configuration and compilation Add httpx as an optional dependency for API functionality
Enhance error handling and user guidance when API configuration fails: - Add more detailed and helpful messages for missing configuration files - Provide step-by-step setup instructions with emojis and links - Improve invalid API key error messages with troubleshooting tips
Provide more helpful information when API rate limit is exceeded, including upgrade options and retry suggestion
Add support for testing API keys directly without saving to config via --api-key flag. Improves debugging workflow by allowing quick key validation and provides better error messaging when no configuration exists.
refactor(config): standardize config file naming to api.toml feat(cli): add support for .pine file compilation in run command docs: update help texts to reflect new config file naming
Update error messages for rate limit exceeded cases to use consistent formatting and emojis across compile and run commands
Move API error handling logic from compile.py and run.py into a new api_error_handler.py module. This improves maintainability by eliminating duplicate error handling code and provides consistent error messages across commands.
- Update type hints to use modern Python syntax (PEP 585/604) - Remove unused imports and optimize import statements - Improve code formatting and docstring consistency - Add static method decorators where appropriate - Enhance error handling and exception management - Update .gitignore with development environment entries - Maintain backward compatibility while modernizing codebase
- Add verify_token_local() method for offline JWT validation without server requests - Support both standard 'exp' and custom 'e' expiration fields in JWT tokens - Add get_usage() and validate_api_key() helper methods to PyneComp class - Enhance compile command to show API usage statistics and key expiration info - Add --usage flag to display statistics after compilation - Remove deprecated api.py CLI command module - Improve error handling with informative user messages
- Added optional sleep to _print_usage to ensure usage statistics are saved before display in compile command. - Improved formatting and clarity of API error messages in api_error_handler. - Removed redundant or outdated lines from command init template.
- Enhance _handle_http_error to accept pre-extracted error messages - Add support for structured error responses in API error handling - Remove emojis and streamline error messages in CLI output - Improve parsing of JSON error responses with detailed error info
Add comprehensive documentation for the new `compile` command that converts Pine Script to Python using PyneSys API. Update related CLI documentation files to reflect Pine Script support in the `run` command and add cross-references. Update README.md to include basic Pine Script usage examples and add new compile.md documentation file with detailed usage instructions, examples, and troubleshooting information.
Extract error condition checks into descriptive variables for better clarity Use consistent formatting for multi-line strings and conditions Add missing import for regex operations
- Updated Discord invite link in pyproject.toml, README.md, and docs. - Removed references to free Discord API keys in README.md. - Updated PineComp description for clarity in ecosystem docs. - Reformatted some long lines for improved readability.
Implement DataConverter class to automatically detect and convert CSV/TXT/JSON files to OHLCV format when needed. Includes symbol/timeframe auto-detection from filenames and TOML symbol info generation. The conversion is now integrated into the run command, eliminating manual conversion steps. Add support for --symbol and --timeframe parameters in run command for non-OHLCV files. Improve error handling and user feedback during conversion process. Include progress indicators for better UX.
- Add AUTO timeframe option and auto-detection logic - Make symbol and timeframe optional with default values - Improve help text to show default behaviors - Move data path expansion before conversion logic
The timeframe option was removed as it's no longer needed for data conversion. The conversion logic now always uses "AUTO" for timeframe detection.
Add file existence check and automatic path resolution for data files in convert_from command Simplify script path handling in run command by removing redundant extension check
wallneradam
commented
Aug 7, 2025
Contributor
Author
wallneradam
left a comment
wallneradam
left a comment
There was a problem hiding this comment.
Also please merge main into this repo, because there were a lot of changes in the main repository.
- Add auto-detect symbol from filename functionality - Enhance timeframe validation with TV-compatible formats - Improve OHLCV file detection and validation - Refactor symbol info generation using SymInfo class - Remove STRUCT_FORMAT from __all__ in ohlcv_file.py
Major enhancements: - Add comprehensive trading hours detection and analysis in OHLCVWriter - Implement advanced tick size detection with histogram clustering - Add interval auto-correction for files starting with gaps - Extend symbol and provider detection for more filename formats OHLCVWriter improvements: - New trading hours collection and analysis system - Samples existing files on open to detect trading patterns - Smart data sufficiency checks based on timeframe - Detects 24/7 (crypto) vs business hours patterns - Advanced tick size detection with confidence scoring - Automatic interval correction when smaller intervals detected DataConverter updates: - Rename symbol_from_filename to guess_symbol_from_filename (public API) - Rename _detect_symbol_type to guess_symbol_type (public API) - Enhanced filename pattern matching for various formats - Improved provider detection from filename patterns - Better handling of complex patterns like ccxt_BYBIT_BTC_USDT_USDT CLI improvements: - Update data.py and run.py to use new guess_symbol_from_filename API - Remove AUTO timeframe option, simplify validation - Better error handling and user feedback Bug fixes: - Fix timeframe.from_seconds() missing minute format output - Remove redundant symbol type validation Tests: - Add comprehensive DataConverter tests - Add symbol type detection tests - Add timeframe from_seconds tests - Add trading hours detection tests in OHLCV file
… config - Update convert-to command description to clarify OHLCV file usage - Add details on automatic extension handling and output file naming - Document auto-detection of symbol, provider, and format from filenames - Provide filename pattern examples for symbol/provider extraction - Explain TOML config generation: symbol type, tick size, trading hours, interval - Update run command docs to support auto conversion from CSV/JSON/TXT - Add section on advanced analysis during conversion (tick size, hours, etc.) - Clarify supported formats and filename detection for run command - Improve examples for automatic conversion and config generation
Implement TXT file support with auto-detected delimiters (tab, semicolon, pipe) Add type checking for price data collection to handle NA values Improve timestamp handling in CSV/TXT parsing with better error handling
…iles The csv module was replaced with a custom parser implementation to have better control over the parsing process and handle edge cases more reliably. This change improves error handling and supports more flexible input formats while maintaining the same functionality.
…de duplication - Created standalone _parse_timestamp() function to centralize timestamp parsing - Eliminates ~80 lines of duplicated code across load_from_csv, load_from_txt, and load_from_json - Also renamed format_float to _format_float and _combine_tick_estimates_no_gcd to _combine_tick_estimates for consistency
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.