Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scanner.c: multiple definition of is_newline - trying to compile all tree-sitter grammars #55

Open
milahu opened this issue Feb 20, 2024 · 2 comments · May be fixed by #65
Open

scanner.c: multiple definition of is_newline - trying to compile all tree-sitter grammars #55

milahu opened this issue Feb 20, 2024 · 2 comments · May be fixed by #65

Comments

@milahu
Copy link

milahu commented Feb 20, 2024 •
edited
Loading

i just tried to use this to compile all tree-sitter grammars

but the build fails with many multiple definition errors
so it seems like a bad idea to compile all grammars into one binary

tree-sitter-languages.nix 
{ lib
, python3
, fetchFromGitHub
, tree-sitter-grammars
}:

/*
# debug: build faster
let old-tree-sitter-grammars = tree-sitter-grammars; in
let
  tree-sitter-grammars = {
    tree-sitter-html = old-tree-sitter-grammars.tree-sitter-html;
  };
in
*/

python3.pkgs.buildPythonPackage rec {
  pname = "tree-sitter-languages";
  version = "1.10.2";
  pyproject = true;

  src = fetchFromGitHub {
    owner = "grantjenks";
    repo = "py-tree-sitter-languages";
    rev = "v${version}";
    hash = "sha256-AuPK15xtLiQx6N2OATVJFecsL8k3pOagrWu1GascbwM=";
  };

  buildInputs = [
    python3.pkgs.cython
  ];

  nativeBuildInputs = [
    python3.pkgs.setuptools
    python3.pkgs.wheel
  ];

  propagatedBuildInputs = [
    python3.pkgs.tree-sitter
  ];

  postUnpack = ''
    cd $sourceRoot
    mkdir vendor
    ${
      builtins.concatStringsSep "" (
        builtins.attrValues (
          builtins.mapAttrs
          (n: p:
            "ln -v -s ${p.src.outPath} vendor/${n}\n"
          )
          (lib.filterAttrs (k: v: v ? src) tree-sitter-grammars)
        )
      )
    }
    cd ..
  '';

  postBuild = ''
    echo creating $out/${python3.sitePackages}/tree_sitter_languages/languages.so

    repo_paths=(
    ${
      builtins.concatStringsSep "" (
        builtins.attrValues (
          builtins.mapAttrs
          (n: p:
            "  'vendor/${n}'\n"
          )
          (lib.filterAttrs (k: v: v ? src) tree-sitter-grammars)
        )
      )
    }
    )
    # get actual repo paths
    # fix: No such file or directory: 'vendor/tree-sitter-markdown/src/parser.c
    for idx in ''${!repo_paths[@]}; do
      dir=''${repo_paths[$idx]}
      [ -e $dir/src/parser.c ] && continue
      parser=$(find $dir -path '*/src/parser.c')
      dir=''${parser%/src/parser.c}
      repo_paths[$idx]=$dir
    done

    #mkdir -p $out/${python3.sitePackages}/tree_sitter_languages
    build_py=$(
      echo "import tree_sitter"
      echo "repo_paths = ["
      for dir in ''${repo_paths[@]}; do
        echo "  '$dir',"
      done
      echo "]"
      echo "output_path = '$out/${python3.sitePackages}/tree_sitter_languages/languages.so'"
      echo "tree_sitter.Language.build_library(output_path, repo_paths)"
    )
    echo "$build_py" | grep -n "" # debug
    python3 -c "$build_py"
  '';

  pythonImportsCheck = [ "tree_sitter_languages" ];

  meta = with lib; {
    description = "Python module with all tree-sitter languages";
    homepage = "https://github.com/grantjenks/py-tree-sitter-languages";
    license = licenses.asl20;
    maintainers = with maintainers; [ ];
  };
}
build.py 
import tree_sitter
repo_paths = [
  'vendor/tree-sitter-bash',
  'vendor/tree-sitter-beancount',
  'vendor/tree-sitter-bibtex',
  'vendor/tree-sitter-bitbake',
  'vendor/tree-sitter-c',
  'vendor/tree-sitter-c-sharp',
  'vendor/tree-sitter-clojure',
  'vendor/tree-sitter-cmake',
  'vendor/tree-sitter-comment',
  'vendor/tree-sitter-commonlisp',
  'vendor/tree-sitter-cpp',
  'vendor/tree-sitter-css',
  'vendor/tree-sitter-cuda',
  'vendor/tree-sitter-cue',
  'vendor/tree-sitter-dart',
  'vendor/tree-sitter-devicetree',
  'vendor/tree-sitter-dockerfile',
  'vendor/tree-sitter-dot',
  'vendor/tree-sitter-eex',
  'vendor/tree-sitter-elisp',
  'vendor/tree-sitter-elixir',
  'vendor/tree-sitter-elm',
  'vendor/tree-sitter-embedded-template',
  'vendor/tree-sitter-erlang',
  'vendor/tree-sitter-fennel',
  'vendor/tree-sitter-fish',
  'vendor/tree-sitter-fortran',
  'vendor/tree-sitter-gdscript',
  'vendor/tree-sitter-glimmer',
  'vendor/tree-sitter-glsl',
  'vendor/tree-sitter-go',
  'vendor/tree-sitter-godot-resource',
  'vendor/tree-sitter-gomod',
  'vendor/tree-sitter-gowork',
  'vendor/tree-sitter-graphql',
  'vendor/tree-sitter-haskell',
  'vendor/tree-sitter-hcl',
  'vendor/tree-sitter-heex',
  'vendor/tree-sitter-hjson',
  'vendor/tree-sitter-html',
  'vendor/tree-sitter-http',
  'vendor/tree-sitter-janet-simple',
  'vendor/tree-sitter-java',
  'vendor/tree-sitter-javascript',
  'vendor/tree-sitter-jsdoc',
  'vendor/tree-sitter-json',
  'vendor/tree-sitter-json5',
  'vendor/tree-sitter-jsonnet',
  'vendor/tree-sitter-julia',
  'vendor/tree-sitter-just',
  'vendor/tree-sitter-kotlin',
  'vendor/tree-sitter-latex',
  'vendor/tree-sitter-ledger',
  'vendor/tree-sitter-llvm',
  'vendor/tree-sitter-lua',
  'vendor/tree-sitter-make',
  'vendor/tree-sitter-nickel',
  'vendor/tree-sitter-nix',
  'vendor/tree-sitter-norg',
  'vendor/tree-sitter-norg-meta',
  'vendor/tree-sitter-nu',
  'vendor/tree-sitter-org-nvim',
  'vendor/tree-sitter-perl',
  'vendor/tree-sitter-pgn',
  'vendor/tree-sitter-php',
  'vendor/tree-sitter-pioasm',
  'vendor/tree-sitter-prisma',
  'vendor/tree-sitter-proto',
  'vendor/tree-sitter-pug',
  'vendor/tree-sitter-python',
  'vendor/tree-sitter-ql',
  'vendor/tree-sitter-ql-dbscheme',
  'vendor/tree-sitter-query',
  'vendor/tree-sitter-r',
  'vendor/tree-sitter-regex',
  'vendor/tree-sitter-rego',
  'vendor/tree-sitter-rst',
  'vendor/tree-sitter-ruby',
  'vendor/tree-sitter-rust',
  'vendor/tree-sitter-scala',
  'vendor/tree-sitter-scheme',
  'vendor/tree-sitter-scss',
  'vendor/tree-sitter-smithy',
  'vendor/tree-sitter-solidity',
  'vendor/tree-sitter-sparql',
  'vendor/tree-sitter-sql',
  'vendor/tree-sitter-supercollider',
  'vendor/tree-sitter-surface',
  'vendor/tree-sitter-svelte',
  'vendor/tree-sitter-tiger',
  'vendor/tree-sitter-tlaplus',
  'vendor/tree-sitter-toml',
  'vendor/tree-sitter-tsq',
  'vendor/tree-sitter-turtle',
  'vendor/tree-sitter-typst',
  'vendor/tree-sitter-uiua',
  'vendor/tree-sitter-verilog',
  'vendor/tree-sitter-vim',
  'vendor/tree-sitter-vue',
  'vendor/tree-sitter-wgsl',
  'vendor/tree-sitter-yaml',
  'vendor/tree-sitter-yang',
  'vendor/tree-sitter-zig',
]
output_path = '/nix/store/87v1hw8y1y82010g66fm3b6qq7v4av3p-python3.11-tree-sitter-languages-1.10.2/lib/python3.11/site-packages/tree_sitter_languages/languages.so'
tree_sitter.Language.build_library(output_path, repo_paths)
build.log 
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-rst/src/scanner.o: in function `is_newline':
scanner.c:(.text+0x190): multiple definition of `is_newline'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-comment/src/scanner.o:scanner.c:(.text+0x20): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-rst/src/scanner.o: in function `is_space':
scanner.c:(.text+0x1b0): multiple definition of `is_space'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-comment/src/scanner.o:scanner.c:(.text+0x40): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-svelte/src/scanner.o: in function `can_contain':
scanner.c:(.text+0xbd0): multiple definition of `can_contain'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0x1f0): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-svelte/src/scanner.o: in function `serialize':
scanner.c:(.text+0x2150): multiple definition of `serialize'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-org-nvim/src/scanner.o:scanner.c:(.text+0x0): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-svelte/src/scanner.o: in function `deserialize':
scanner.c:(.text+0x2250): multiple definition of `deserialize'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-org-nvim/src/scanner.o:scanner.c:(.text+0x150): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-svelte/src/scanner.o: in function `scan':
scanner.c:(.text+0x2bc0): multiple definition of `scan'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-org-nvim/src/scanner.o:scanner.c:(.text+0x700): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-vue/src/scanner.o: in function `tree_sitter_html_external_scanner_create':
scanner.cc:(.text+0xaf0): multiple definition of `tree_sitter_html_external_scanner_create'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0x5b0): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-vue/src/scanner.o: in function `tree_sitter_html_external_scanner_serialize':
scanner.cc:(.text+0xb20): multiple definition of `tree_sitter_html_external_scanner_serialize'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0xac0): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-vue/src/scanner.o: in function `tree_sitter_html_external_scanner_destroy.localalias':
scanner.cc:(.text+0xc40): multiple definition of `tree_sitter_html_external_scanner_destroy'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0xf00): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-vue/src/scanner.o: in function `tree_sitter_html_external_scanner_deserialize':
scanner.cc:(.text+0xe20): multiple definition of `tree_sitter_html_external_scanner_deserialize'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0xba0): first defined here
/nix/store/xdqs45iclhp9dz8zz9pvn5zivjbhid1a-binutils-2.40/bin/ld: /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-vue/src/scanner.o: in function `tree_sitter_html_external_scanner_scan':
scanner.cc:(.text+0x1850): multiple definition of `tree_sitter_html_external_scanner_scan'; /build/tmpxyj0x25atree_sitter_language/vendor/tree-sitter-html/src/scanner.o:scanner.c:(.text+0x5c0): first defined here
collect2: error: ld returned 1 exit status
@milahu
Copy link
Author

milahu commented Feb 20, 2024 •
edited
Loading

out of scope

these are bugs in the parsers

see also
tree-sitter/tree-sitter-html#64
https://arduino.stackexchange.com/questions/45419/multiple-definition-of-local-variable-in-linking

@milahu milahu closed this as completed Feb 20, 2024
This was referenced Feb 20, 2024
Open
Closed
Open
@milahu
Copy link
Author

milahu commented Feb 20, 2024

these are bugs in the parsers

but still, these errors can be avoided by compiling each parser to a separate binary
this also has the benefit that we can reuse parser binaries

see also milahu/nur-packages@85dccc2

@milahu milahu reopened this Feb 20, 2024
@ObserverOfTime ObserverOfTime linked a pull request May 18, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Rewrite for py-tree-sitter 0.22 ObserverOfTime/py-tree-sitter-languages
1 participant
@milahu