-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
109 additions
and
98 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Examples | ||
=== | ||
|
||
- [Tests](https://github.com/YieldLang/yieldlang/tree/main/tests) - See the tests for more examples. | ||
- [Visualization of syntax tree](https://github.com/YieldLang/yieldlang/discussions/17) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,97 +1,17 @@ | ||
Overview | ||
======== | ||
|
||
YieldLang is a [meta-language](https://en.wikipedia.org/wiki/Metalanguage) for LLMs to process (produce and parse) structured info. | ||
```{toctree} | ||
:maxdepth: 2 | ||
:hidden: | ||
- View our [publications](publications.md) for more information | ||
|
||
## Simple Usage | ||
|
||
```bash | ||
pip install yieldlang | ||
``` | ||
|
||
Import the `TextGenerator` class and define a generator. The `top` method always serves as the entry point for the generator. You can treat the generator as an iterator and use a `for` loop to iterate over the generated text. For example: | ||
|
||
```py | ||
from yieldlang import TextGenerator | ||
|
||
class G(TextGenerator): | ||
def top(self): | ||
yield "Hello, World!" | ||
|
||
for text in G(): | ||
print(text) | ||
``` | ||
|
||
Set another sampler for the generator (default is random sampling). For example, set the large language model sampler: | ||
|
||
```py | ||
sampler = MyLLMSampler() | ||
print(list(G(sampler))) | ||
``` | ||
|
||
Use combinators (e.g., `select`, `repeat`, `join`, etc.) to define grammar rules in the `TextGenerator`. For example, for JSON values: | ||
|
||
|
||
```py | ||
def value(self): | ||
yield select( | ||
self.object, | ||
self.array, | ||
self.string, | ||
self.number, | ||
self.boolean, | ||
self.null | ||
) | ||
``` | ||
|
||
This is equivalent to the EBNF form: | ||
|
||
```ebnf | ||
value = object | ||
| array | ||
| string | ||
| number | ||
| boolean | ||
| null | ||
``` | ||
|
||
Generate a sequence easily. For example: | ||
|
||
```py | ||
def array(self): | ||
yield select( | ||
('[', self.ws, ']'), | ||
('[', self.elements, ']') | ||
) | ||
``` | ||
|
||
You can get the string just generated and add branches, loops, and other control structures to the generation rules. For example: | ||
|
||
```py | ||
def diagram(self): | ||
match (yield self.diagram_type): | ||
case "flowchart": | ||
yield self.flowchart | ||
case "gannt": | ||
yield self.gannt | ||
``` | ||
|
||
Use a loop statement in the generator. For example: | ||
|
||
```py | ||
def repeat4(self, s): | ||
l = [] | ||
for _ in range(4): | ||
l.append((yield s)) | ||
self.do_my_own_thing(l) | ||
installation | ||
usages/simple | ||
examples/index | ||
``` | ||
|
||
Print the generated context tree (convertible to an abstract syntax tree): | ||
YieldLang is a [meta-language](https://en.wikipedia.org/wiki/Metalanguage) for LLMs to process (produce and parse) structured info. View our [publications](publications.md) for more information. | ||
|
||
```py | ||
def print_context_tree(): | ||
ctx = yield from G() | ||
print(ctx) | ||
``` | ||
- 🧠 Based on a coroutine generator and sampler architecture | ||
- 🤖 Stream-sends characters and parses the context above into a syntax tree | ||
- 🦾 Build formal grammars with classes, methods, and combinators |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,90 @@ | ||
# Simple Usage | ||
|
||
```bash | ||
pip install yieldlang | ||
``` | ||
|
||
Import the `TextGenerator` class and define a generator. The `top` method always serves as the entry point for the generator. You can treat the generator as an iterator and use a `for` loop to iterate over the generated text. For example: | ||
|
||
```py | ||
from yieldlang import TextGenerator | ||
|
||
class G(TextGenerator): | ||
def top(self): | ||
yield "Hello, World!" | ||
|
||
for text in G(): | ||
print(text) | ||
``` | ||
|
||
Set another sampler for the generator (default is random sampling). For example, set the large language model sampler: | ||
|
||
```py | ||
sampler = MyLLMSampler() | ||
print(list(G(sampler))) | ||
``` | ||
|
||
Use combinators (e.g., `select`, `repeat`, `join`, etc.) to define grammar rules in the `TextGenerator`. For example, for JSON values: | ||
|
||
|
||
```py | ||
def value(self): | ||
yield select( | ||
self.object, | ||
self.array, | ||
self.string, | ||
self.number, | ||
self.boolean, | ||
self.null | ||
) | ||
``` | ||
|
||
This is equivalent to the EBNF form: | ||
|
||
```ebnf | ||
value = object | ||
| array | ||
| string | ||
| number | ||
| boolean | ||
| null | ||
``` | ||
|
||
Generate a sequence easily. For example: | ||
|
||
```py | ||
def array(self): | ||
yield select( | ||
('[', self.ws, ']'), | ||
('[', self.elements, ']') | ||
) | ||
``` | ||
|
||
You can get the string just generated and add branches, loops, and other control structures to the generation rules. For example: | ||
|
||
```py | ||
def diagram(self): | ||
match (yield self.diagram_type): | ||
case "flowchart": | ||
yield self.flowchart | ||
case "gannt": | ||
yield self.gannt | ||
``` | ||
|
||
Use a loop statement in the generator. For example: | ||
|
||
```py | ||
def repeat4(self, s): | ||
l: list[str] = [] | ||
for _ in range(4): | ||
l.append((yield s)) | ||
self.do_my_own_thing(l) | ||
``` | ||
|
||
Print the generated context tree (convertible to an abstract syntax tree): | ||
|
||
```py | ||
def print_context_tree(): | ||
ctx = yield from G() | ||
print(ctx) | ||
``` |