Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port to LoongArch64 #257

Closed
wants to merge 2 commits into from
Closed

Port to LoongArch64 #257

wants to merge 2 commits into from

Conversation

merore
Copy link

@merore merore commented Aug 23, 2021

[root@9d30c71260ba greenlet]# python -m unittest discover -v greenlet.tests
test_break_ctxvars (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_assignment_different_thread (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_assignment_while_running (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_not_propagated (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_propagated_by_context_run (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_propagated_by_setting_attribute (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_context_shared (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_not_broken_if_using_attribute_instead_of_context_run (greenlet.tests.test_contextvars.ContextVarsTests) ... skipped 'ContextVar not supported'
test_contextvars_errors (greenlet.tests.test_contextvars.NoContextVarsTests) ... ok
test_exception_switch (greenlet.tests.test_cpp.CPPTests) ... ok
test_getcurrent (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_new_greenlet (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_dead (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_error (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_setparent (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_switch (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_switch_kwargs (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_throw (greenlet.tests.test_extension_interface.CAPITests) ... ok
test_circular_greenlet (greenlet.tests.test_gc.GCTests) ... ok
test_dead_circular_ref (greenlet.tests.test_gc.GCTests) ... ok
test_finalizer_crash (greenlet.tests.test_gc.GCTests) ... ok
test_inactive_ref (greenlet.tests.test_gc.GCTests) ... ok
test_generator (greenlet.tests.test_generator.GeneratorTests) ... ok
test_genlet_bad (greenlet.tests.test_generator_nested.NestedGeneratorTests) ... ok
test_genlet_simple (greenlet.tests.test_generator_nested.NestedGeneratorTests) ... ok
test_layered_genlets (greenlet.tests.test_generator_nested.NestedGeneratorTests) ... ok
test_nested_genlets (greenlet.tests.test_generator_nested.NestedGeneratorTests) ... ok
test_permutations (greenlet.tests.test_generator_nested.NestedGeneratorTests) ... ok
test_abstract_subclasses (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_dealloc (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_dealloc_other_thread (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_dealloc_switch_args_not_lost (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_deepcopy (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_exc_state (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_exception (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_frame (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_implicit_parent_with_threads (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_instance_dict (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_issue_245_reference_counting_subclass_no_threads (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_issue_245_reference_counting_subclass_threads (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_parent_equals_None (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_parent_restored_on_kill (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_parent_return_failure (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_recursive_startup (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_run_equals_None (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_send_exception (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_simple (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_switch_kwargs (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_switch_kwargs_to_parent (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_switch_to_another_thread (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_thread_bug (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_threaded_reparent (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_threaded_updatecurrent (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_threads (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_throw_doesnt_crash (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_throw_exception_not_lost (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_tuple_subclass (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_two_children (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_two_recursive_children (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_unexpected_reparenting (greenlet.tests.test_greenlet.TestGreenlet) ... ok
test_dead (greenlet.tests.test_greenlet.TestRepr) ... ok
test_formatting_produces_native_str (greenlet.tests.test_greenlet.TestRepr) ... ok
test_initial (greenlet.tests.test_greenlet.TestRepr) ... ok
test_main_from_other_thread (greenlet.tests.test_greenlet.TestRepr) ... ok
test_main_in_background (greenlet.tests.test_greenlet.TestRepr) ... ok
test_main_while_running (greenlet.tests.test_greenlet.TestRepr) ... ok
test_arg_refs (greenlet.tests.test_leaks.TestLeaks) ... ok
test_issue251_issue252_need_to_collect_in_background (greenlet.tests.test_leaks.TestLeaks) ... expected failure
test_issue251_killing_cross_thread_leaks_list (greenlet.tests.test_leaks.TestLeaks) ... ok
test_kwarg_refs (greenlet.tests.test_leaks.TestLeaks) ... ok
test_threaded_adv_leak (greenlet.tests.test_leaks.TestLeaks) ... ok
test_threaded_leak (greenlet.tests.test_leaks.TestLeaks) ... ok
test_stack_saved (greenlet.tests.test_stack_saved.Test) ... ok
test_class (greenlet.tests.test_throw.ThrowTests) ... ok
test_kill (greenlet.tests.test_throw.ThrowTests) ... ok
test_throw_goes_to_original_parent (greenlet.tests.test_throw.ThrowTests) ... ok
test_val (greenlet.tests.test_throw.ThrowTests) ... ok
test_exception_disables_tracing (greenlet.tests.test_tracing.TracingTests) ... ok
test_greenlet_tracing (greenlet.tests.test_tracing.TracingTests) ... ok
test_version (greenlet.tests.test_version.VersionTests) ... ok
test_dead_weakref (greenlet.tests.test_weakref.WeakRefTests) ... ok
test_dealloc_weakref (greenlet.tests.test_weakref.WeakRefTests) ... ok
test_inactive_weakref (greenlet.tests.test_weakref.WeakRefTests) ... ok

----------------------------------------------------------------------
Ran 83 tests in 1.615s

OK (skipped=8, expected failures=1)

[root@9d30c71260ba greenlet]# uname -a
Linux 9d30c71260ba 4.19.190-1.lns8.loongarch64 #1 SMP Tue Jun 29 10:22:01 CST 2021 loongarch64 loongarch64 loongarch64 GNU/Linux

@merore
Copy link
Author

merore commented Aug 23, 2021

This is the manual for the LoongArch64.
http://loongson.cn/images/pdf/LoongArch/1.pdf

@jamadden
Copy link
Contributor

My understanding is that this is a brand new (only announced a few months ago) proprietary CPU, generally being produced for the Chinese domestic market.

Have you verified this on actual hardware, or just an emulator? How would I go about verifying this (presumably in…some... emulator with…some...Linux distro? I haven't looked, but I don't think I've even heard of Linux distros supporting this new CPU yet)?

@merore
Copy link
Author

merore commented Aug 31, 2021

@jamadden
Thank your for you replay ! You are rigth, this is not only a new CPU but also a new ISA.

The linux kernel is also processing, at the same time we build our OS base on linux for this CPU. So when building the OS, we are also actively seeking support from the upstream community.

Finally, I run these on a physical machine, because both the machine and OS are available. If you are interested in this ISA, you can also try it.

@jamadden
Copy link
Contributor

jamadden commented Aug 31, 2021

you can also try it.

That is my question: How do I try it? Which open source Linux distribution do I install in which open source emulator in order to try it? I need step-by-step instructions, please.

My understanding is that neither the kernel nor userspace has been released (upstreamed) yet. That suggests that they are still subject to change, which means this patch can quickly bit rot.

#define STACK_MAGIC 0

#define REGS_TO_SAVE "s0", "s1", "s2", "s3", "s4", "s5", \
"s6", "s7", "s8", "fp", \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure fp can be included? I see this in my independent port before I saw your PR:

loongarch64-unknown-linux-gnu-gcc -fPIC -I/home/xenon/greenlet-venv/include -I/usr/include/python3.9 -c src/greenlet/greenlet.c -o build/temp.linux-loongarch64-3.9/src/greenlet/greenlet.o
In file included from src/greenlet/slp_platformselect.h:59,
                 from src/greenlet/greenlet.c:370:
src/greenlet/platform/switch_loongarch64_unix.h: In function ‘slp_switch’:
src/greenlet/platform/switch_loongarch64_unix.h:31:1: error: $r22 cannot be used in ‘asm’ here
   31 | }
      | ^
error: command '/usr/bin/loongarch64-unknown-linux-gnu-gcc' failed with exit code 1

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am sure fp can be included. I think your error message is caused by gcc compilation optimization. This is my compilation flags.

loongarch64-loongson-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c src/greenlet/greenlet.c -o build/temp.linux-loongarch64-3.6/src/greenlet/greenlet.o

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's not good for portability, you can't force one particular set of CFLAGS, due to different distros just doing their job differently. I know that fp/s9 actually should be saved, but how do we force it in this case? I'm using the latest upstream submission for my toolchain and kernel, and a self-compiled Gentoo userland; let me know if you need the Gentoo tarball to reproduce the issue.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I can reproduce this issue in other way. This is indeed a problem.

Inspired by your question, I found this the setup.py. I think we should add appropriate compilation optimization flags for loongarch64 here.

if ((sys.platform == "openbsd4" and os.uname()[-1] == "i386")
    or ("-with-redhat-3." in platform.platform() and platform.machine() == 'i686')
    or (sys.platform == "sunos5" and os.uname()[-1] == "sun4v")
    or ("SunOS" in platform.platform() and platform.machine() == "sun4v")
    or (sys.platform == "linux" and platform.machine() == "ppc")):
    os.environ["CFLAGS"] = ("%s %s" % (os.environ.get("CFLAGS", ""), "-Os")).lstrip()

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimization flags are not the proper way to solve this, in my opinion, because they tend to depend on intricate compiler internals to work. The "solution" will likely be fragile and break when the stars don't align.

Actually, the way most other projects manually save/restore preserved registers is to write concrete asm fragments, even whole .S files; doing this via clobber list is something I have never seen before, yet in this project every port does that. How about noticing your compiler team about this so they can provide some input? @-mentioning them here should be enough.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@merore 所以这里正确的处理方式是什么?python setup.py bdist_wheel是一样的报上面的错

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Duanexiao Please refer to the latest commit.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Duanexiao Please refer to the latest commit.

gevent项目我看贵司还没提兼容loongarch64的代码?但是龙芯的pypi里面有这个

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gevent项目我看贵司还没提兼容loongarch64的代码?但是龙芯的pypi里面有这个

code here https://github.com/merore/greenlet/commits/master

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gevent项目我看贵司还没提兼容loongarch64的代码?但是龙芯的pypi里面有这个

code here https://github.com/merore/greenlet/commits/master

我说的是gevent项目

xen0n added a commit to xen0n/gentoo that referenced this pull request Apr 30, 2022
The LoongArch port for greenlet hasn't been merged yet [1], so disable
for now.

[1]: python-greenlet/greenlet#257

Signed-off-by: WANG Xuerui <xen0n@gentoo.org>
xen0n added a commit to xen0n/gentoo that referenced this pull request Apr 30, 2022
The LoongArch port for greenlet hasn't been merged yet [1], so disable
for now.

[1]: python-greenlet/greenlet#257

Signed-off-by: WANG Xuerui <xen0n@gentoo.org>
xen0n added a commit to xen0n/gentoo that referenced this pull request May 1, 2022
The LoongArch port for greenlet hasn't been merged yet [1], so disable
for now.

[1]: python-greenlet/greenlet#257

Signed-off-by: WANG Xuerui <xen0n@gentoo.org>
@merore
Copy link
Author

merore commented May 18, 2023

@jamadden sorry to bother you again.
Linux already supports loongarch64, but the operating system is not perfect. So I am building a docker image that includes loongarch64 rootfs and qemu-loongarch64 to run on x86.

vt-alt pushed a commit to altlinux/specs that referenced this pull request Aug 7, 2023
- Upstream version 2.0.2. Amongst other things solves FTBFS with GCC 13.
- Support LoongArch architecture (from
  python-greenlet/greenlet#257)
@jamadden jamadden closed this in f1005b9 Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants