A unified, intelligent log querying and correlation engine for the command line. Think of it as a mini Splunk/Datadog/ELK stack that searches events, not files.
- Multi-format support: Auto-detects and parses Nginx, JSON/JSONL, Syslog (BSD & RFC5424)
- Powerful query language: SQL-like syntax with filtering, grouping, and aggregations
- Multi-cloud ready: AWS CloudWatch, GCP Logging, Azure Monitor, Kubernetes
- Perl-powered transformations: Full Perl expressions for custom field manipulation
- Multiple output formats: Table, JSON, CSV with syntax highlighting
- Perl 5.36 or later
- mise (recommended) or manual Perl installation
# Clone the repository
git clone https://github.com/yourusername/perl_pro_text.git
cd perl_pro_text
# Install dependencies
cpanm --installdeps .
# Verify installation
perl -Ilib bin/sift --help# Query JSON logs from stdin
echo '{"level":"error","status":500,"message":"Connection failed"}' | sift query 'status >= 500'
# Query nginx access logs
cat /var/log/nginx/access.log | sift query 'status >= 400'
# Filter with AND/OR conditions
cat app.log | sift query 'status >= 500 and service == "auth"'
# Use IN expressions
cat app.log | sift query 'status in {500, 502, 503}'
# Group and aggregate
cat access.log | sift query 'status >= 400 group by ip count'
# Output as JSON
cat app.log | sift query 'level == "error"' --output jsonfield == "value" # Equality (strings)
field == 200 # Equality (numbers)
field != "value" # Not equal
field > 100 # Greater than
field >= 100 # Greater than or equal
field < 100 # Less than
field <= 100 # Less than or equal
expr1 and expr2 # Both must be true
expr1 or expr2 # Either must be true
not expr # Negation
(expr1 or expr2) and expr3 # Grouping with parentheses
status in {500, 502, 503}
level in {"error", "critical"}
count # Count matching events
avg field # Average of numeric field
sum field # Sum of numeric field
min field # Minimum value
max field # Maximum value
group by field # Group results
group field # Short form
sort by field asc # Sort ascending
sort field desc # Sort descending
limit 10 # Limit results
status >= 400 and method == "POST"
group by ip
count
sort by count desc
limit 10
| Format | Description | Auto-detected |
|---|---|---|
nginx |
Nginx combined access log and error log | Yes |
json |
JSON Lines (JSONL) structured logs | Yes |
syslog |
BSD and RFC5424 syslog formats | Yes |
| Source | Provider | Command |
|---|---|---|
aws |
AWS CloudWatch Logs | aws logs CLI |
gcp |
Google Cloud Logging | gcloud logging CLI |
azure |
Azure Monitor | az monitor CLI |
k8s |
Kubernetes | kubectl logs |
sift query <expression> # Execute a log query
sift extract # Extract fields with patterns
sift find # Find matching log entries
sift formats # List supported log formats
sift sources # List available log sourcestable(default) - Colored ASCII tablejson- JSON arraycsv- CSV format
sift query 'status >= 500' --output json
sift query 'status >= 500' --output csv# Run tests
prove -l -r t/
# Run specific test
prove -l t/unit/query_executor.t
# Check syntax
perl -c -Ilib lib/Sift/Pro.pmperl_pro_text/
├── bin/
│ └── sift # CLI entry point
├── lib/
│ └── Sift/
│ ├── Pro.pm # Main application
│ ├── CLI.pm # Command-line interface
│ ├── Event.pm # Unified event class
│ ├── Query/ # Query language (Marpa::R2)
│ ├── Parser/ # Log format parsers
│ ├── Source/ # Log sources (file, cloud)
│ └── Output/ # Output formatters
├── t/ # Tests
├── cpanfile # Dependencies
└── Makefile.PL # Build configuration
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please ensure:
- All tests pass (
prove -l -r t/) - Code follows existing style conventions
- New features include tests
This project is licensed under the MIT License - see the LICENSE file for details.
- Marpa::R2 - Powerful BNF parser
- Moo - Minimalist Object Orientation
- Type::Tiny - Type constraints