New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
descriptors (getset, wrapper, method), __new__, __init__, builtin_function_or_method, hack js inheritance to match py #1092
descriptors (getset, wrapper, method), __new__, __init__, builtin_function_or_method, hack js inheritance to match py #1092
Conversation
5d51d05
to
a450657
Compare
Ran speed tests on a version this branch with this branch
master
so approx |
Holy moly that's some improvement @s-cork |
I'll see if I can find some time today to go through this 😄 |
I wouldn't look at this branch! I haven't pushed any changes to this branch since I started hacking too deep! And I went quite deep... It's far from ready for review. But if you're curious about current progress, it can be found here. It's mostly me hacking away with these ideas, but it does present some nice features that I hope will make it into this pr. The branch mostly works - (I'm down to to 24 failing py unit 2 tests...) interesting files to look at:
Once you start implementing descriptors the game changes... And I've had a bit too much time... At this stage I'm most curious about - what you think of the new api... p.s. I wouldn't check the speeds right now. After my day of hacking I lost it somewhere so will need to check that! |
I'm still trying to get my head around all this but I have a few questions:
These are the things that are most exposed to the outside and would be most painful to break. |
In my head i would leave the In fact Which now uses the Similarly the compiler does The new api doesn’t replace the existing api by any means, it is a With Internally we can use it. And anyone who wants to build a My aim with this pr will be to just show the bare bones - which I hope will A benefit with this pr will hopefully be that: and no change to how to build a module. |
Update: I did some more hacking today and my times for
I think this is because I was working on our use of number slots. and on pystone.py (10000 passes avg of 10 loops)
|
That's massive @s-cork. 🎉 🚀 |
update on this branch - I now have a version that's passing tests so I've pushed to this branch. It's still quite rough around the edges... |
This is magnificent stuff -- it makes the type system simultaneously cleaner and more correct, and makes using Skulpt from JS a whole lot easier. I look forward to going over it in more detail. Am I right in thinking this is a breaking change? (At very least, it moves the |
I have yet to document anything but things that break - but the short answer is - yes - it's breaking. Particularly on constructing a builtin e.g. the only valid call to tuple's constructor is: new Sk.buitin.tuple(Array|undefined) Sk.builtin.tuple(Array) -> "assertion fail must be called with 'new'"
new Sk.builtin.tuple(pyIterable) -> "assertion fail must be called with an Array"
/*to convert a pyIterable to a tuple you'd need to use*/
Sk.misceval.callsimArray(Sk.builtin.tuple, [pyIterable]) Typically a constructor should only ever be called with a JS object (this improves the speed and is useful for the compiler which already calls the constructor with the JS object). The one exception is Other breaking changes off the top of my head
(not especially breaking but notable)
I'll try and separate some logic in this pr and create a separate pr for small non breaking changes that can be pre-release changes - e.g. everywhere I found we weren't using |
a very minor detail - related to implementing slots in this pr - in python 3.8:
This seems sensible to me. removes some repeat code in the code base. I'll put this into this pr unless any strong objections. Only changes I think 1 test in our library for multiple inheritance where it should override |
An update on this branch. I've recently been working on int and unifying int and long. I've got it to a place where py2 compatibility is maintained by making I used Things I'm struggling with: I'm doing well with speed. though I have an oddity that if I remove a function on lots left to do... but just in case you felt this was going cold. |
Another update on this branch. I've got a slightly adapted tact for name mangling. You can see the approach in the string constructor. There also seemed to me, no reason to treat mangling any different for names/words so I unified the approach. Speed update
so overall approx |
I can add another success report for this PR. A colleague and I are working an educational Python environment called Pytch, which aims to be a stepping stone between MIT's Scratch and Python. We've built on Skulpt, making heavy use of its suspensions to implement event-triggered concurrency. If you haven't come across Scratch, it's an excellent system for learning programming, where you write programs by dragging/dropping blocks. There's a nice runtime with interactive sprites, built-in graphics, sounds, etc., so it's easy for novice programmers to concentrate on the interesting bits. Once they're skilled in Scratch, people often want to move on to Python. But then not only do they have to deal with a whole new way of expressing their programs (as text, getting all the syntax right), but they also have to deal with a whole new runtime, leaving behind their sprites, sounds, and so on. Pytch lets them stay in that world, keeping all the understanding they've built up about it, and concentrate just on the change from block-based to text-based programming. I made an experimental branch of Pytch, and merged in @s-cork's branch. I had to make a handful of changes to Pytch: using More info:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1:1 with @s-cork, tested, and in production on Anvil for months now!
Hi all, I think it's time to get this sucker merged! I have gone over it with a fairly fine tooth-comb with @s-cork, and all my feedback is already incorporated in the current state of the PR. We have been running it in production at Anvil for most of a year now (which means we've shaken out all the bugs our users could find, and some of them ride Skulpt quite hard). It's getting traction elsewhere, too: At least @acbart's, @bennorth's and @alexwenbj's projects are using it in production. Waiting for it to land is also blocking up Skulpt development - either PRs are stacking on top of it (#1196, #1240, #1241) or others are nervous to contribute to something that's about to be refactored. It's a big win for python compatibility, and the performance gain is simply eye-popping (@rixner should be happy ;) ). It translates beyond benchmarks, too - since we shipped it, we got a lot of "Is it just me, or has everything got noticeably faster?" on the Anvil forums. @bnmnetp I think we should just hit the button, and as soon as possible. |
FWIW I've looked at the getattr and call internals (mostly with an eye on performance gotchas more than correctness) and those looked good to me. |
Trying this out on my Fork. Things looked very promising until I get to document. This simple program:
Which has always worked fine now fails with
What needs to be updated in the definition to get it to work again? In debugging it fails in the Sk.miscval.callSimArray line.. |
Ok, so I've tracked this down a bit further. The error is coming from a line:
where self.checked evaluates to |
Nevermind -- It turns out that I should have been making a new bool not a new string. In my defense this particular line of code is at least 8 years old. But oddly has been working many thousands of times a day until I test merged this PR. 😄 |
And to be fair Merging... |
…ort-post-1092 Partial docstring support (post #1092)
It aims to: close #1043, close #1090, close #1075, close #942, close #1093, close #1084, close #1094, close #819, close #1195
The aim of the pr is to:
type objects
(forcetype
onto their prototypical chains!)getset_descriptor
, 'wrapper_descriptor', 'method_descriptor'mappingproxy
fortype objects
property
,classmethod
,staticmethod
builtin_function_or_method
akaSk.builtin.sk_method
inheritance (type and object and maybe metaclasses)
since the implementation of #1131 all type objects are instances of
Sk.builtin.type
and this pr uses the benefits thatprototypical
inheritances brings.When doing lookups we no longer have to walk the mro for single inherited objects and do a direct access of the property.
Paving the way for metaclasses:
You can now inherit from type. Though meta classes are not supported since
kwargs
are not supported in the compiler forclass B(metaclass=A)
Multiple inheritance
since multiple inheritance breaks protoypical inheritance I set a new flag
sk$prototypical
on all objects.$mroMerge
function.false
we take the slow path in various instances (liketypeLookup
) and check up theMRO
.If this flag is
true
we take the fast path and simply do the equivalent of__new__
and__init__
tp$init
andtp$new
we get to be stricter about how the constructor can be used internally.new
as a conventiontp$call
tp$new
tp$init
methods in order to create an object.new Sk.builtin.float_(new Sk.builtin.int_(1))
is valid becausenb$float
is defined. But passing a python string to the float constructor won't work.getset_descriptors
to implement
getset_descriptors
effectively I move allklass
attributes onto theirprototype
.this improves the logic for
klass.tp$getattr
implementing
getset_descriptors
allows for the removal of__class__
,__name__
,__bases__
,__mro__
,__dict__
from the code base, apart from asgetset_descriptors
on eitherSk.builtin.type
orSk.builtin.object
tp$mro
is now an array. Since this is critical fortypeLookup
it seems sensible.Similarly all
sk$klass
objects get a__dict__
getset_descriptor
which returns the$d
for the instance of thatklass
... removing the need to hack__dict__
mapping proxy
a
mappingproxy
is a non-writeabledict
like object. By implementing amappingproxy
I remove the$d
from alltype
objects. This removes the requirement to look inside the$d
for lookups. When the user asks forklass.__dict__
they hit thetype
getset_descriptor
for__dict__
which gathers up the attributes on theprototype
and returns amapping_proxy
the
mappingproxy
implementation is close to Cpython and can also be used with thetypes
module with tests.Suggested new API for building native types:
A limitation of this API is that you can only have one
base
which is true of all builtins.there are other options that be tagged onto the object literal including:
constructor
: the JS implementation - how to create an instance of the object (defaults tonew Function
)slots
: any slot functions (defining any slot function gives the equivalent dunder method)getsets
:getset_descriptors
e.g.__dict__
methods
: e.g.str.count
classmethods
: e.g.dict.fromkeys
base
: defaults toSk.builtin.object
meta
: defaults toSk.builtin.type
proto
: anything here will be added onto theklass.prototype
slots
norgetsets
normethods
flags
: anything here will be added directly onto theklass
There are a bunch of other changes that are part of this PR
Sk.builtin.Genric*
toSk.generic.*
Sk.generic.new
... etcwrapper_descriptors
Sk.abstr.setUpSlots
builtin_function_or_method
type (Sk.buitlin.sk_method
) used by:method_descriptors
mean the call signature can be defined and separates thefunction.tp$call
api fromsk_method.tp$call
apimath
with the new api - allSk.builtin.func
becomeSk.builtin.sk_method
objects with little fuss.collections
with the new api (including an improvednamedtuple
(close Named Tuple #1040) implementation)itertools
with the new api - these are now all type objects rather than hacked generatorsoperator
with the new api and adding missing features.sk$acceptable_as_base_class
- not all builtins are subclassablesk$object
- a way to determine that you have a python object e.g.Sk.asserts.assert(obj.sk$object)
sk$prototypical
sk$builtinBase
- the most builtin base of an objectsk$baseClass
- a flag on type objects that are direct children ofobject
$
rather than also ending in an_