Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Establishing Iterators #101

Merged
merged 35 commits into from
Mar 3, 2023
Merged

Establishing Iterators #101

merged 35 commits into from
Mar 3, 2023

Conversation

phorward
Copy link
Member

@phorward phorward commented Jan 13, 2023

This PR implements iterators for Tokay.

  • RefValueIter is a trait that implements any iterator
  • Iter is the Tokay iterator value object
  • MethodIter is an iterator working on indexed method calls
  • RangeIter is an iterator for counting integer values
  • MapIter is an iterator with a mapping an iterator to a callable, which also can be used as a filter
  • EnumIter is an iterator mapping an iterator with an index

The following Tokay functions are made available:

  • iter() creates an iterator from an object, by calling object.iter() or automatically a MethodIter on an object
  • iter_rev() reverses an iterator, if possible
  • iter_next() returns the iterators next element
  • iter_collect() collects and iterator into a list; is used by list() on an iter as well
  • iter_enum() creates an enumerated iterator
  • iter_map() creates a mapped/filtered iterator
  • range() creates a range counting iterator

The implementation is only a place to start. There are still some restrictions and problems discussed in PR #101.
There's also the need for aggregation functions as described in issue #88.


What's already working:

for i in (1,2,3) print(i)

d = (first => 1, second => 2, third => 3)
for k in d print(k)
for k in d.keys print(k)
for k in d.items print(k)

 # create lists from iters
print(list(iter(d)))
print(list(d.keys))

Still existing problems:

  • Efficiency?
  • range() function likewise Python works without a target object, just on a min-max-step construct
  • How to implement dict.keys() or dict.items() efficiently?
  • What about custom iterator behaviors, as one cannot inherit from iter; Tokay is NOT an object-oriented language.
  • What about reversable iterators?
  • Iterator-Functions
    • len() - counts the iterator
    • map() - maps the iterator to a function; using void paradigm acts like a filter.
    • enum() - returns a list of [index, item] for each entry
    • collect() - collect iterator into a list
    • min(), max(), sum(), avg() - aggregate functions on iterators (will be separately done with Provide iterator aggregation functions min(), max(), sum(), avg() #88)

'f' was used twice; This wasn't a problem, its just personal favor.
This already allows for this
```
i = iter((1,2,3))
i.next  # 1
i.next  # 2
i.next  # 3
i.next  # void
```
@phorward phorward added feature New feature or request help wanted Extra attention is needed tokay labels Jan 13, 2023
@phorward phorward self-assigned this Jan 13, 2023
This allows for unnested
```
for i in (1,2,3) print(i)
for c in "hello" print(c)
for d in ("a" => 23, 2) print(d)
```
This is essential for `for`-loops over iterators.
Now using a parametrizable version of Iter, which allows to specifiy a method and an operation for the iterator's get and increment.
- MethodOpIter is the previous Iter
- Builtin for any `dyn Iterator<Item = RefValue>`
- `range` builtin iterator
Incomplete, currently only next or prev can be called on an iterator manually.

IMHO this isn't the best choice. I think it's better to define this via a trait, and a `rev()` there turns the iterator into a reverted version, which counts down.
I'm still not sure if this is the right way.
- RefValueIter is a trait for Iterators on RefValue
- Iter is an Iterator-implementing container for RefValueIter
- MethodIter and RangeIter implement RefValueIter

rev() and repr() unfinished yet.
Invalidates the iterator befor its first iteration in case of invalid values.
Using dyn_clone crate.
Makes use of dyn_clone-crate obsolete again (trying to keep dependencies low)
- Iter, RefValueIter and BoxedRefValueIter remain in values::iter
- MethodIter moved into values::methoditer
- RangerIter moved into builtin::range
- Fixed rev for RangeIter
- Bad hack for str to use negative indexing feature
@phorward phorward marked this pull request as ready for review February 25, 2023 23:49
@phorward
Copy link
Member Author

Ok I think Iterators are now in a state to be merged into Tokay.

@phorward phorward changed the title Iterators Establishing Iterators Feb 26, 2023
@phorward phorward marked this pull request as draft February 26, 2023 11:01
@phorward
Copy link
Member Author

Ok I think Iterators are now in a state to be merged into Tokay.

After some consideration, let's finish iterators by providing some more functionality.

@phorward
Copy link
Member Author

#88 refers to this

Required for further developments.

- call() to call an object with optional Context
- call_direct() to call an object with defined Context
Well... this thing was - and still is - nasty.

This state does allow to call any callable from a builtin.
And implements a mapped iterator by the way.

Here, this works:

	for i in range(2, 100).map(@x { print(x) x*x }) print(i)

Anyway, I'm not sure if this is the best way.

iter_map() generates another iter, which is borrowed when the provided function is executed. This leads into the problem, that in case map makes use of the same iterator, program will crash. This isn't useful.
Improving build.rs/build.tok to accept impl blocks.
@phorward
Copy link
Member Author

phorward commented Mar 3, 2023

The iterator feature in the currently started version with the mapping unfortunately seems to be a bottomless pit, since several problems arise:

  1. due to the possibility of using functions as callbacks, the context must necessarily be passed and passed through, which is partially implemented here.
  2. since the iterator is borrowed mutable a further access possibly within a callback to the same iterator object leads inevitably to a panic.
  3. in general the passing of the context/args/nargs parameter configuration (or even if parameters are already on the stack) should be done in the form of a separate struct or enum, but this entails major rebuilding work in the entire code.

It would therefore be better if the map feature - possibly even using a yield function - were implemented in pure VM code.

Nevertheless, this branch contains important and necessary changes, which is why it should be remembered. Whether the map feature will be removed again or not remains open at the moment.

@phorward phorward marked this pull request as ready for review March 3, 2023 22:01
@phorward phorward merged commit 0f1d00c into tokay-lang:main Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant