Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix type of evaluated_value on string to allow bytes #721

Merged
merged 2 commits into from
May 26, 2023

Conversation

ljodal
Copy link
Contributor

@ljodal ljodal commented Jul 8, 2022

Summary

This can return bytes if the string is a bytestring, e.g.:

In [1]: import libcst as cst

In [2]: cst.parse_expression('b"foo"').evaluated_value
Out[2]: b'foo'

@facebook-github-bot
Copy link

Hi @ljodal!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

@ljodal ljodal changed the title Fix type of `evaluated_value on string Fix type of evaluated_value on string Jul 8, 2022
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 8, 2022
@facebook-github-bot
Copy link

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@zsol
Copy link
Member

zsol commented Jul 9, 2022

Thanks for the PR; this seems reasonable to me but let's fix the type errors this introduces before merging. There are instructions in the readme about how you can set this up on your local machine without relying on CI.

@ljodal
Copy link
Contributor Author

ljodal commented Jul 9, 2022

I tried to set everything up locally, but I got some kind of json decode error when installing some of the dev dependencies (left my laptop at the office, so I don't have the error in from of me). I'm on an M1 Mac, so might be something there? Anyway, I can try again after the weekend :)

@ljodal
Copy link
Contributor Author

ljodal commented Jul 11, 2022

I'm unable to get pyre working locally. As mentioned I'm on an M1 Mac and there seems to be a bunch of problems with the project setup for that architecture. First I hit facebook/pyre-check#560 because pyre-check is pinned to 0.9.9. Tried updating it to both 0.9.10 and the latest release, but then I'm just getting this facebook/pyre-check#614 (even if I install from source).

Running pip install -r requirements.txt -r requirements-dev.txt gives me an error as well when trying to compile maturin. I'm not familiar with rust, so I have no idea why:

Compile error
        running "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-arch" "arm64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-gfull" "-DNDEBUG" "-c" "-o/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-install-5x3nuxd5/maturin_cb3d7abe64a345d0940b511b57000ddd/target/debug/build/ring-076f384d9a02721e/out/aesv8-armx-linux64.o" "/Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S"
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:18:17: error: unexpected token in '.section' directive
        .section .rodata
                        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:28:1: error: unknown directive
        .hidden GFp_aes_hw_set_encrypt_key
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:29:1: error: unknown directive
        .type GFp_aes_hw_set_encrypt_key,%function
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:161:1: error: unknown directive
        .size GFp_aes_hw_set_encrypt_key,.-GFp_aes_hw_set_encrypt_key
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:163:1: error: unknown directive
        .hidden GFp_aes_hw_encrypt
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:164:1: error: unknown directive
        .type GFp_aes_hw_encrypt,%function
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:191:1: error: unknown directive
        .size GFp_aes_hw_encrypt,.-GFp_aes_hw_encrypt
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:193:1: error: unknown directive
        .hidden GFp_aes_hw_decrypt
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:194:1: error: unknown directive
        .type GFp_aes_hw_decrypt,%function
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:221:1: error: unknown directive
        .size GFp_aes_hw_decrypt,.-GFp_aes_hw_decrypt
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:223:1: error: unknown directive
        .hidden GFp_aes_hw_ctr32_encrypt_blocks
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:224:1: error: unknown directive
        .type GFp_aes_hw_ctr32_encrypt_blocks,%function
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:403:1: error: unknown directive
        .size GFp_aes_hw_ctr32_encrypt_blocks,.-GFp_aes_hw_ctr32_encrypt_blocks
        ^
        /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/pregenerated/aesv8-armx-linux64.S:407:19: error: unexpected token in '.section' directive
        .section .note.GNU-stack,"",%progbits
                          ^
        thread 'main' panicked at 'execution failed', /Users/sigurd/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.15/build.rs:664:9
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      warning: build failed, waiting for other jobs to finish...
      Traceback (most recent call last):
        File "/Users/sigurd/Developer/libcst/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/sigurd/Developer/libcst/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/sigurd/Developer/libcst/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-build-env-lps_nf4j/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 208, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-build-env-lps_nf4j/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 194, in _build_with_temp_dir
          self.run_setup()
        File "/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-build-env-lps_nf4j/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 142, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 81, in <module>
          setup(
        File "/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-build-env-lps_nf4j/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 145, in setup
          return distutils.core.setup(**attrs)
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/q2/bg3_vyg15m1g3w7d4wfrk9km0000gn/T/pip-build-env-lps_nf4j/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 228, in run
          self.run_command('install')
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "setup.py", line 58, in run
          subprocess.check_call(
        File "/opt/homebrew/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cargo', 'rustc', '--bin', 'maturin', '--', '-C', 'link-arg=-s']' returned non-zero exit status 101.

I could try to fix the errors, but not super keen on doing that without type checker support. Let me know if you'd rather I just open an issue?

@zsol
Copy link
Member

zsol commented Jul 12, 2022

Let me see if I can get you some help for that Pyre issue.

About the maturin build problem: that's interesting, I've never seen that issue before (admittedly, I don't have an arm64 macbook to test on). based on this thread maybe you have an x64 rust toolchain installed as the default on your arm64 machine? You can check by running rustc -vV | grep host:

@ljodal
Copy link
Contributor Author

ljodal commented Jul 12, 2022

Yeah, my entire tool chain is native arm64. Doing pip install maturin works fine, so maybe it’s trying to use an old version without arm64 wheels when I install from requirements?

This can return bytes if the string is a bytestring, e.g.:

    In [1]: import libcst as cst

    In [2]: cst.parse_expression('b"foo"').evaluated_value
    Out[2]: b'foo'
@ljodal
Copy link
Contributor Author

ljodal commented Dec 3, 2022

@zsol I took a second stab at this now and got pyre 0.9.10 running locally, so I've fixed the errors I got with that (hopefully it should be all)

return left_val + right_val
if isinstance(left_val, str) and isinstance(right_val, str):
return left_val + right_val
return None
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely sure what to do with mixed values here, but I assume that's invalid Python and that returning None should be fine?

@ljodal
Copy link
Contributor Author

ljodal commented May 25, 2023

@zsol Any chance of a review of this? Would be nice to get this into one of the upcoming releases :)

Copy link
Member

@zsol zsol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, and sorry for taking this long :)

@@ -1700,6 +1700,8 @@ def get_docstring_impl(
evaluated_value = val.evaluated_value
else:
return None
if isinstance(evaluated_value, bytes):
return None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to look up the specs for docstrings. Looks like they are guaranteed to be string literals (as opposed to bytes literals), so this is fine (although I personally would've written if not isinstance(evaluated_value, str))!

@zsol zsol changed the title Fix type of evaluated_value on string Fix type of evaluated_value on string to allow bytes May 26, 2023
@zsol zsol merged commit a594fe1 into Instagram:main May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants