Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor logos-derive from tree to graph #94

Merged
merged 101 commits into from
Mar 30, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
45227f8
Starting the refactor
maciejhirsz Mar 21, 2020
bb3834f
Debug prints and testing helpers
maciejhirsz Mar 22, 2020
45ffd32
Disabled tests that are unwanted for now
maciejhirsz Mar 22, 2020
aab6f95
Even better errors
maciejhirsz Mar 22, 2020
7dcebd5
Organizing the graph module
maciejhirsz Mar 22, 2020
3ce0e1a
Start building the graph from attributes
maciejhirsz Mar 22, 2020
d2d0bfa
Deduplicate some code
maciejhirsz Mar 22, 2020
57e9aa1
Adding logic to the new Fork
maciejhirsz Mar 22, 2020
56692aa
Range iterator
maciejhirsz Mar 22, 2020
f500c65
Forks should only match 1 byte, Sequences instead of Branches
maciejhirsz Mar 22, 2020
e1b5bcb
Rename Sequence to Rope, impl Rope::fork_off(&self) -> Fork
maciejhirsz Mar 22, 2020
2a36be6
Fix comment typo
maciejhirsz Mar 22, 2020
68c25ec
Rearrange lut and miss in Fork
maciejhirsz Mar 22, 2020
c3d77b8
No reason not to own the Token Ident
maciejhirsz Mar 22, 2020
cf8e950
Convert #[token] attributes to Ropes
maciejhirsz Mar 22, 2020
e175128
Non-conflict merging done
maciejhirsz Mar 22, 2020
395baad
Rope to only use Vec<u8>
maciejhirsz Mar 23, 2020
768ac1e
Handle branches for non-regex tokens
maciejhirsz Mar 23, 2020
6f29c63
Reimplementing regex
maciejhirsz Mar 23, 2020
9422e97
Basic Regex handling
maciejhirsz Mar 23, 2020
9273cf7
Handle *, +, and ? regex repeats
maciejhirsz Mar 23, 2020
09013dc
Return empty fork on empty HirKind
maciejhirsz Mar 23, 2020
fa6261e
Cleaner HirKind::Concat parsing
maciejhirsz Mar 23, 2020
ae93032
Fix looping for OneOrMore repeats
maciejhirsz Mar 23, 2020
eb23db8
Don't allow regex patterns that could match empty strings
maciejhirsz Mar 24, 2020
f5bd143
Faster merge cache lookups
maciejhirsz Mar 24, 2020
4cb8cf0
Commented out all unused code, fixed warnings
maciejhirsz Mar 24, 2020
41e4319
Group concats in regex, better error handling
maciejhirsz Mar 24, 2020
5dab886
Regex alternations, remove old regex code
maciejhirsz Mar 24, 2020
413ca12
Remove old tree code
maciejhirsz Mar 24, 2020
60f27a7
Working utf8 sequences, Ropes need to expand into ranges
maciejhirsz Mar 24, 2020
a36ac71
Shrinking the ungodly \w
maciejhirsz Mar 24, 2020
34d3494
Tree shaking
maciejhirsz Mar 24, 2020
3d1ad0e
Concatenate ranges into Ropes
maciejhirsz Mar 24, 2020
7c40a58
Fix the + loops!
maciejhirsz Mar 24, 2020
05046e1
Flatten Node & NodeBody into just Node
maciejhirsz Mar 25, 2020
7c6a947
Panic when casting non-ascii ClassUnicodeRange to Range
maciejhirsz Mar 25, 2020
414f11e
Finding merging loops
maciejhirsz Mar 25, 2020
8fbc300
Finding merging loops
maciejhirsz Mar 25, 2020
f2b3c60
Merging looping nodes together
maciejhirsz Mar 25, 2020
0414653
Rename Rope::fork_off to Rope::into_fork, take owned `self`
maciejhirsz Mar 25, 2020
617ef8b
Tweaking node merges
maciejhirsz Mar 25, 2020
39ca3b3
DRY merging ropes with other nodes
maciejhirsz Mar 26, 2020
cac3d3a
Merging loops with ropes
maciejhirsz Mar 26, 2020
b9f720e
Correctly propagate miss value on rope splitting
maciejhirsz Mar 26, 2020
9c57665
Debug print forks with curly barces
maciejhirsz Mar 26, 2020
898be29
Token disambiguation
maciejhirsz Mar 26, 2020
a66dc28
Improved disambiguation for regex
maciejhirsz Mar 26, 2020
02370e6
Remove dead code
maciejhirsz Mar 26, 2020
6e07985
Remove `Lexicon` from the `Logos` trait
maciejhirsz Mar 26, 2020
d3bca51
Generating an empty impl
maciejhirsz Mar 26, 2020
e6d6635
Tests restored, compiling, and failing
maciejhirsz Mar 26, 2020
2ceae31
Restored the binary tests
maciejhirsz Mar 26, 2020
8db4c1e
Generator scaffolding
maciejhirsz Mar 26, 2020
d95607f
First crude implementation of the generator
maciejhirsz Mar 26, 2020
b2a979c
One passing test :)
maciejhirsz Mar 26, 2020
7d61e51
Use lex.error() for errors
maciejhirsz Mar 27, 2020
c0ac36d
Rename Token to Leaf
maciejhirsz Mar 27, 2020
5ff5c32
Turn Leaf into an enum to support trivia
maciejhirsz Mar 27, 2020
ed80d87
Jump to root on trivia
maciejhirsz Mar 27, 2020
a44594e
Implemented trivia
maciejhirsz Mar 27, 2020
88bd559
Covering more tests and performance
maciejhirsz Mar 27, 2020
9cfc71c
Lazy generator
maciejhirsz Mar 27, 2020
3546fb4
Pass down Context when generating code
maciejhirsz Mar 27, 2020
5d3ac9f
Keep track of the stack when generating
maciejhirsz Mar 27, 2020
01c5408
Unwinding self-looping forks
maciejhirsz Mar 27, 2020
02c5c73
Working on Context passing
maciejhirsz Mar 28, 2020
9ed97e4
Passing miss value via Context
maciejhirsz Mar 28, 2020
b93b7f9
Packing the graph
maciejhirsz Mar 28, 2020
59f8eb7
More passing tests
maciejhirsz Mar 28, 2020
262f100
Fix regex alternation, floats are passing
maciejhirsz Mar 28, 2020
6a8508b
All tests but strings passing
maciejhirsz Mar 28, 2020
3d5d44c
Progress on strings
maciejhirsz Mar 28, 2020
94563a7
Fix Meta::loop_entry_from
maciejhirsz Mar 28, 2020
733a84e
Tweaking error handling
maciejhirsz Mar 28, 2020
5997dba
All advaned tests are passing :D
maciejhirsz Mar 28, 2020
9d6a0a6
Avoid .reserve() in regex parsing when possible
maciejhirsz Mar 28, 2020
13d3731
Recursive drain of dangling misses from regex
maciejhirsz Mar 28, 2020
7f17722
Merge miss values on repeats
maciejhirsz Mar 28, 2020
15935e9
Splitting fallback and miss
maciejhirsz Mar 28, 2020
95cbd4c
Functional context juggling, tests mostly green!
maciejhirsz Mar 28, 2020
9183626
Re-enable binary tests
maciejhirsz Mar 28, 2020
5f3574e
Functional callbacks
maciejhirsz Mar 29, 2020
c3657fb
All green
maciejhirsz Mar 29, 2020
e014eb8
Add a panic if two empty nodes are ever tried to be merged
maciejhirsz Mar 29, 2020
d7b3ed9
Introduced Mir to regex
maciejhirsz Mar 29, 2020
7420972
Reintroduce fast loop
maciejhirsz Mar 29, 2020
ec8c74c
Lookup tables for complex ranges
maciejhirsz Mar 29, 2020
acd64cc
Drop old generator, reformat _fast_loop
maciejhirsz Mar 29, 2020
406de2f
Hir to Mir with TryFrom
maciejhirsz Mar 30, 2020
e5e8bd2
Break up Generator into modules
maciejhirsz Mar 30, 2020
4bd0e96
Move _fast_loop into fork where it belongs
maciejhirsz Mar 30, 2020
e9dde8a
Faster forks
maciejhirsz Mar 30, 2020
0d15d35
Jump tables
maciejhirsz Mar 30, 2020
1551f3e
Print unsuffixed u8s in jump tables
maciejhirsz Mar 30, 2020
82d8f49
Calculate minimum amount of bytes to read on forks
maciejhirsz Mar 30, 2020
1a23356
Relax discriminants, generic lex.read() for multiple bytes on forks
maciejhirsz Mar 30, 2020
ef90fe7
Move Context into its own module
maciejhirsz Mar 30, 2020
07775ea
Laying down foundation for passing bound check bytes
maciejhirsz Mar 30, 2020
583a105
Passing pre-bound-check arrays across jumps
maciejhirsz Mar 30, 2020
4e2efbc
Version bump!
maciejhirsz Mar 30, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 99 additions & 30 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ members = [
[profile]
release = { lto = true }
bench = { lto = true }

[patch.crates-io]
logos-derive = { path = "./logos-derive" }
13 changes: 9 additions & 4 deletions logos-derive/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "logos-derive"
version = "0.10.0-rc2"
version = "0.10.0-rc3"
authors = ["maciejhirsz <maciej.hirsz@gmail.com>"]
license = "MIT/Apache-2.0"
description = "Create ridiculously fast Lexers"
Expand All @@ -16,9 +16,14 @@ name = "logos_derive"
proc-macro = true

[dependencies]
beef = "0.4.1"
fnv = "1.0.6"
rustc-hash = "1.0.1"
syn = { version = "0.15", features = ["extra-traits", "full"] }
quote = "0.6"
proc-macro2 = "0.4"
syn = { version = "1.0.17", features = ["extra-traits", "full"] }
quote = "1.0.3"
proc-macro2 = "1.0.9"
regex-syntax = "0.6"
utf8-ranges = "1.0"

[dev-dependencies]
pretty_assertions = "0.6.1"
73 changes: 73 additions & 0 deletions logos-derive/src/error.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
use std::fmt;

use beef::lean::Cow;
use proc_macro2::{Span, TokenStream};
use quote::{quote_spanned, ToTokens, TokenStreamExt};

pub type Result<T> = std::result::Result<T, Error>;

pub struct Error(Cow<'static, str>);

#[derive(Debug)]
pub struct SpannedError {
message: Cow<'static, str>,
span: Span,
}

impl Error {
pub fn new<M>(message: M) -> Self
where
M: Into<Cow<'static, str>>,
{
Error(message.into())
}

pub fn span(self, span: Span) -> SpannedError {
SpannedError {
message: self.0,
span,
}
}
}

impl fmt::Display for Error {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
self.0.fmt(f)
}
}

impl fmt::Debug for Error {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
fmt::Display::fmt(self, f)
}
}

impl From<regex_syntax::Error> for Error {
fn from(err: regex_syntax::Error) -> Error {
Error(err.to_string().into())
}
}

impl From<&'static str> for Error {
fn from(err: &'static str) -> Error {
Error(err.into())
}
}

impl From<String> for Error {
fn from(err: String) -> Error {
Error(err.into())
}
}

impl ToTokens for SpannedError {
fn to_tokens(&self, tokens: &mut TokenStream) {
let message = &*self.message;

tokens.append_all(
quote_spanned!(self.span => {
compile_error!(#message)
})
)
}
}
Loading