rgw/lua: allow read access to object data #47719

yuvalif · 2022-08-21T17:56:34Z

Signed-off-by: Yuval Lifshitz ylifshit@redhat.com

this PR is to reintroduce the change from this PR: #46550
that was reverted in: #47612

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

anthonyeleven

Docs LGTM; we hashed these out in the previous PR. I did make one comment suggesting an additional word.

Approval is only for docs, not for code.

anthonyeleven · 2022-08-22T06:35:59Z

doc/radosgw/lua-scripting.rst


-By default, all lua standard libraries are available in the script, however, in order to allow for other lua modules to be used in the script, we support adding packages to an allowlist:
+ - ``prerequest`` which will execute a script before each operation is performed
+ - ``postrequest`` which will execute after each operation is performed


Suggest s/operation/RGW operation/

cbodley · 2022-08-22T21:33:49Z

src/rgw/rgw_common.h

+  rgw::lua::Background* lua_background = nullptr;
+  rgw::sal::LuaManager* lua_manager = nullptr;
+


yuvalif · 2022-08-25T16:19:32Z

tethology results: http://pulpito.front.sepia.ceph.com/yuvalif-2022-08-22_03:34:48-rgw-wip-yuval-lua-filter2-distro-default-smithi/

know barbican issue: https://tracker.ceph.com/issues/54247
"Failed to fetch package version"
known issues with multiste
know tempest issues: https://tracker.ceph.com/issues/57266
multiple test_librgw_file.sh failures

cbodley · 2022-08-25T16:28:06Z

2022-08-22T20:05:37.658 INFO:tasks.workunit.client.0.smithi039.stdout:[==========] Running 24 tests from 1 test suite.
2022-08-22T20:05:37.658 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] Global test environment set-up.
2022-08-22T20:05:37.659 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] 24 tests from LibRGW
2022-08-22T20:05:37.659 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.INIT
2022-08-22T20:05:37.669 INFO:tasks.workunit.client.0.smithi039.stderr:ignoring --setuser ceph since I am not root
2022-08-22T20:05:37.670 INFO:tasks.workunit.client.0.smithi039.stderr:ignoring --setgroup ceph since I am not root
2022-08-22T20:05:38.017 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.INIT (360 ms)
2022-08-22T20:05:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.MOUNT_NOROOT
2022-08-22T20:05:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.MOUNT_NOROOT (1 ms)
2022-08-22T20:05:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.MOUNT
2022-08-22T20:05:38.020 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.MOUNT (0 ms)
2022-08-22T20:05:38.020 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.SETUP_HIER1
2022-08-22T20:05:38.024 INFO:tasks.workunit.client.0.smithi039.stdout:creating: nfsroot:foo/bar/baz/quux
2022-08-22T20:05:38.025 INFO:tasks.workunit.client.0.smithi039.stderr:*** Caught signal (Segmentation fault) **
2022-08-22T20:05:38.026 INFO:tasks.workunit.client.0.smithi039.stderr: in thread 7fc1dde9b640 thread_name:ceph_test_librg
2022-08-22T20:05:38.027 INFO:tasks.workunit.client.0.smithi039.stderr: ceph version 17.0.0-14421-ge1051899 (e1051899e4ee96e6a8bd281f070ccd6f52b7d296) quincy (dev)
2022-08-22T20:05:38.027 INFO:tasks.workunit.client.0.smithi039.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fc1e764dcf0]
2022-08-22T20:05:38.028 INFO:tasks.workunit.client.0.smithi039.stderr: 2: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x44) [0x7fc1e8da0d04]
2022-08-22T20:05:38.028 INFO:tasks.workunit.client.0.smithi039.stderr: 3: (RGWPutObj::get_lua_filter(std::unique_ptr<rgw::sal::DataProcessor, std::default_delete<rgw::sal::DataProcessor> >*, rgw::sal::DataProcessor*)+0x76) [0x7fc1e8b2e686]
2022-08-22T20:05:38.029 INFO:tasks.workunit.client.0.smithi039.stderr: 4: (RGWPutObj::execute(optional_yield)+0xdd4) [0x7fc1e8b2fbe4]
2022-08-22T20:05:38.029 INFO:tasks.workunit.client.0.smithi039.stderr: 5: (rgw::RGWLibProcess::process_request(rgw::RGWLibRequest*, rgw::RGWLibIO*)+0xad9) [0x7fc1e8a2f8d9]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 6: (rgw::RGWLibProcess::process_request(rgw::RGWLibRequest*)+0x47) [0x7fc1e8a303b7]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 7: (LibRGW_SETUP_HIER1_Test::TestBody()+0x9a2) [0x55fc5fabae52]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 8: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x8f) [0x55fc5fb00d5f]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 9: (testing::Test::Run()+0xd6) [0x55fc5faf4596]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 10: (testing::TestInfo::Run()+0x135) [0x55fc5faf46f5]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 11: (testing::TestSuite::Run()+0xc9) [0x55fc5faf4ba9]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 12: (testing::internal::UnitTestImpl::RunAllTests()+0x445) [0x55fc5faf5195]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 13: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x87) [0x55fc5fb01327]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 14: (testing::UnitTest::Run()+0xa1) [0x55fc5faf47c1]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 15: main()
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 16: __libc_start_main()
2022-08-22T20:05:38.033 INFO:tasks.workunit.client.0.smithi039.stderr: 17: _start()

yuvalif · 2022-08-25T16:30:59Z

2022-08-22T20:05:37.658 INFO:tasks.workunit.client.0.smithi039.stdout:[==========] Running 24 tests from 1 test suite.
2022-08-22T20:05:37.658 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] Global test environment set-up.
2022-08-22T20:05:37.659 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] 24 tests from LibRGW
2022-08-22T20:05:37.659 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.INIT
2022-08-22T20:05:37.669 INFO:tasks.workunit.client.0.smithi039.stderr:ignoring --setuser ceph since I am not root
2022-08-22T20:05:37.670 INFO:tasks.workunit.client.0.smithi039.stderr:ignoring --setgroup ceph since I am not root
2022-08-22T20:05:38.017 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.INIT (360 ms)
2022-08-22T20:05:38.018 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.MOUNT_NOROOT
2022-08-22T20:05:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.MOUNT_NOROOT (1 ms)
2022-08-22T20:05:38.019 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.MOUNT
2022-08-22T20:05:38.020 INFO:tasks.workunit.client.0.smithi039.stdout:[       OK ] LibRGW.MOUNT (0 ms)
2022-08-22T20:05:38.020 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN      ] LibRGW.SETUP_HIER1
2022-08-22T20:05:38.024 INFO:tasks.workunit.client.0.smithi039.stdout:creating: nfsroot:foo/bar/baz/quux
2022-08-22T20:05:38.025 INFO:tasks.workunit.client.0.smithi039.stderr:*** Caught signal (Segmentation fault) **
2022-08-22T20:05:38.026 INFO:tasks.workunit.client.0.smithi039.stderr: in thread 7fc1dde9b640 thread_name:ceph_test_librg
2022-08-22T20:05:38.027 INFO:tasks.workunit.client.0.smithi039.stderr: ceph version 17.0.0-14421-ge1051899 (e1051899e4ee96e6a8bd281f070ccd6f52b7d296) quincy (dev)
2022-08-22T20:05:38.027 INFO:tasks.workunit.client.0.smithi039.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fc1e764dcf0]
2022-08-22T20:05:38.028 INFO:tasks.workunit.client.0.smithi039.stderr: 2: (rgw::lua::read_script(DoutPrefixProvider const*, rgw::sal::LuaManager*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, optional_yield, rgw::lua::context, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x44) [0x7fc1e8da0d04]
2022-08-22T20:05:38.028 INFO:tasks.workunit.client.0.smithi039.stderr: 3: (RGWPutObj::get_lua_filter(std::unique_ptr<rgw::sal::DataProcessor, std::default_delete<rgw::sal::DataProcessor> >*, rgw::sal::DataProcessor*)+0x76) [0x7fc1e8b2e686]
2022-08-22T20:05:38.029 INFO:tasks.workunit.client.0.smithi039.stderr: 4: (RGWPutObj::execute(optional_yield)+0xdd4) [0x7fc1e8b2fbe4]
2022-08-22T20:05:38.029 INFO:tasks.workunit.client.0.smithi039.stderr: 5: (rgw::RGWLibProcess::process_request(rgw::RGWLibRequest*, rgw::RGWLibIO*)+0xad9) [0x7fc1e8a2f8d9]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 6: (rgw::RGWLibProcess::process_request(rgw::RGWLibRequest*)+0x47) [0x7fc1e8a303b7]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 7: (LibRGW_SETUP_HIER1_Test::TestBody()+0x9a2) [0x55fc5fabae52]
2022-08-22T20:05:38.030 INFO:tasks.workunit.client.0.smithi039.stderr: 8: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x8f) [0x55fc5fb00d5f]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 9: (testing::Test::Run()+0xd6) [0x55fc5faf4596]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 10: (testing::TestInfo::Run()+0x135) [0x55fc5faf46f5]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 11: (testing::TestSuite::Run()+0xc9) [0x55fc5faf4ba9]
2022-08-22T20:05:38.031 INFO:tasks.workunit.client.0.smithi039.stderr: 12: (testing::internal::UnitTestImpl::RunAllTests()+0x445) [0x55fc5faf5195]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 13: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x87) [0x55fc5fb01327]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 14: (testing::UnitTest::Run()+0xa1) [0x55fc5faf47c1]
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 15: main()
2022-08-22T20:05:38.032 INFO:tasks.workunit.client.0.smithi039.stderr: 16: __libc_start_main()
2022-08-22T20:05:38.033 INFO:tasks.workunit.client.0.smithi039.stderr: 17: _start()

will add a check for a null lua manager for the UTs

yuvalif · 2022-09-01T12:45:38Z

teuthology results: http://pulpito.front.sepia.ceph.com/yuvalif-2022-08-30_18:21:43-rgw-wip-yuval-lua-filter2-distro-default-smithi/

"Cannot create secret" - expected
multisite - expected
__throw_bad_variant_access crashes - expected
full sync crash - http://qa-proxy.ceph.com/teuthology/yuvalif-2022-08-30_18:21:43-rgw-wip-yuval-lua-filter2-distro-default-smithi/7001818/teuthology.log (new, but seen elsewhere)
No module named 'tasks.... issues - looks like a test issue
failed 'sudo yum -y install cephfs-java'

cbodley · 2022-09-01T12:49:52Z

No module named 'tasks.... issues - looks like a test issue

failed 'sudo yum -y install cephfs-java'

i scheduled a --rerun, hoping these failures go away

$ teuthology-suite -m smithi -c wip-yuval-lua-filter2 --rerun yuvalif-2022-08-30_18:21:43-rgw-wip-yuval-lua-filter2-distro-default-smithi -p 75

pending in https://pulpito.ceph.com/cbodley-2022-09-01_12:48:08-rgw-wip-yuval-lua-filter2-distro-default-smithi

cbodley · 2022-09-01T15:31:15Z

looks like they passed the second time around 👍

cbodley · 2022-09-05T16:33:03Z

i'm seeing a bunch of those test_librgw_file crashes in a recent run, https://pulpito.ceph.com/cbodley-2022-09-04_17:54:41-rgw-wip-cbodley-testing-distro-default-smithi/

the only 2 prs in that batch were #47905 and #47907

i schedule a run against main and see them there too https://pulpito.ceph.com/cbodley-2022-09-05_15:20:23-rgw-main-distro-default-smithi/

edit: opened https://tracker.ceph.com/issues/57441

yuvalif · 2022-09-05T18:11:00Z

@cbodley this is being fixed here: #47964
the issue was introduced here: a1e21d0
please note there is another tracker for this: https://tracker.ceph.com/issues/57418

cbodley · 2022-09-05T18:18:59Z

ty!

rgw/lua: allow read access to object data

e105189

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

yuvalif added documentation rgw tests labels Aug 21, 2022

yuvalif requested a review from cbodley August 21, 2022 17:56

yuvalif requested a review from a team as a code owner August 21, 2022 17:56

github-actions bot added the build/ops label Aug 21, 2022

anthonyeleven approved these changes Aug 22, 2022

View reviewed changes

cbodley approved these changes Aug 22, 2022

View reviewed changes

cbodley added the needs-qa label Aug 22, 2022

cbodley merged commit b45daa6 into ceph:main Sep 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rgw/lua: allow read access to object data #47719

rgw/lua: allow read access to object data #47719

yuvalif commented Aug 21, 2022

anthonyeleven left a comment

anthonyeleven Aug 22, 2022

cbodley Aug 22, 2022

yuvalif commented Aug 25, 2022

cbodley commented Aug 25, 2022

yuvalif commented Aug 25, 2022

yuvalif commented Sep 1, 2022

cbodley commented Sep 1, 2022

cbodley commented Sep 1, 2022

cbodley commented Sep 5, 2022 •

edited

yuvalif commented Sep 5, 2022 •

edited

cbodley commented Sep 5, 2022

		rgw::lua::Background* lua_background = nullptr;
		rgw::sal::LuaManager* lua_manager = nullptr;

rgw/lua: allow read access to object data #47719

rgw/lua: allow read access to object data #47719

Conversation

yuvalif commented Aug 21, 2022

Checklist

anthonyeleven left a comment

Choose a reason for hiding this comment

anthonyeleven Aug 22, 2022

Choose a reason for hiding this comment

cbodley Aug 22, 2022

Choose a reason for hiding this comment

yuvalif commented Aug 25, 2022

cbodley commented Aug 25, 2022

yuvalif commented Aug 25, 2022

yuvalif commented Sep 1, 2022

cbodley commented Sep 1, 2022

cbodley commented Sep 1, 2022

cbodley commented Sep 5, 2022 • edited

yuvalif commented Sep 5, 2022 • edited

cbodley commented Sep 5, 2022

cbodley commented Sep 5, 2022 •

edited

yuvalif commented Sep 5, 2022 •

edited