Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VIP: vyper-json #1520

Closed
alcuadrado opened this issue Jul 8, 2019 · 17 comments · Fixed by #1594
Closed

VIP: vyper-json #1520

alcuadrado opened this issue Jul 8, 2019 · 17 comments · Fixed by #1594
Labels

Comments

@alcuadrado
Copy link

@alcuadrado alcuadrado commented Jul 8, 2019

Simple Summary

Vyper should provide an interface for non-python-based tools to use.

Abstract

This VIP proposes the addition of another CLI tool: vyper-json. This tool will compile contracts, generating a structured JSON output containing everything needed to integrate Vyper, including errors.

Motivation

See #1492

Specification

This section describes the input and output interfaces of vyper-json.

Input description

vyper-json doesn't need any special input apart from a list of files to compile.

An example execution could be vyper-json path/to/contract1.vy path/to/other.vy

Output description

vyper-json outputs a single JSON object with one key of type PATH per input
file, associated with an object of type OUTPUT.

PATH values should be absolute paths. If symlinks are present, they should not
be resolved, keeping the path closer to the user's input.

OUTPUT values are objects with the following properties:

  • files: An array of PATH values containing all the source files used to compile this contract. This is useful to implement things like cache, watchers, etc.
  • errors: An array of ERROR objects. One per each fatal compilation error. This array must be empty if the compilation succeeded.
  • warnings: An array of ERROR objects. One per each compilation warning. This array must be empty if no warning is emitted.
  • output: An object containing bytecode, bytecode_runtime, abi, source_map, method_identifiers as returned by vyper when format combined_json is used. This value should be null if the compilation failed.

ERROR objects must contain the following properties:

  • file: A PATH value pointing to the file where the warning or error originated.
  • line: The error/warning's line number.
  • column: The error/warning's line column.
  • message: An explanation of the warning/error.

Example

Executing vyper-json path/to/contract1.vy path/to/other.vy

should output something like:

{
  "/absolute/path/to/contract1.vy": {
    "files": ["/absolute/path/to/contract1.vy", "/absolute/path/to/interface.vy"],
    "errors": [
      {
        "file": "/absolute/path/to/contract1.vy",
        "line": 1,
        "column": 20,
        "message": "A fatal error"
      }
    ],
    "warnings": [],
    "output": null
  },
  "/absolute/path/to/other.vy": {
    "files": ["/absolute/path/to/other.vy", "/path/path/to/ERC20.vy"],
    "errors": [],
    "warnings": [
      {
        "file": "/absolute/path/to/other.vy",
        "line": 2,
        "column": 30,
        "message": "A warning"
      }
    ],
    "output": { /* same as combined_json */ }
  }
}

Backwards Compatibility

This VIP proposes the introduction of a new command-line tool, which should have no impact on the existing one.

Dependencies

This VIP has no dependencies.

Copyright

Copyright and related rights waived via CC0

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Jul 8, 2019

I like it, did good work specifying this. IIRC solc's "json mode" requires the input to be a JSON string too. This would make it very easy to integrate with JavaScript tooling and web API endpoints.

@alcuadrado

This comment has been minimized.

Copy link
Author

@alcuadrado alcuadrado commented Jul 8, 2019

Yes, solidity's json mode expects a json containing everything it needs as input.

The problem with this approach is that in order to generate that json you need to parse the sources to get their imports and resolve them yourself. That's very problematic. Parsing correctly is not an easy task, and not having a standard imports resolution mechanism can lead to fragmentation in the tooling ecosystem.

@alcuadrado

This comment has been minimized.

Copy link
Author

@alcuadrado alcuadrado commented Jul 19, 2019

Some more info about Solidity's json input.

This last few days I've been working with the Etherscan's contract verification API and UI, and found that they now support using the standard json input. If it were supported in the API, that would solve lots of problems, as it contains all the info needed to reproduce a build. The key advantage is that it's self-contained in a single file.

My proposal for vyper-json doesn't have this property.

I still think that making imports resolution a responsibility of 3rd party tools is very problematic. Maybe have an official tool or vyper-json working mode that generates a self-contained input would solve both problems.

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Jul 19, 2019

Hmm, yeah maybe if you supply all the sources via JSON input, you can have the compiler reference via those strings instead of loading file I/O. Handle it in the script

@alcuadrado

This comment has been minimized.

Copy link
Author

@alcuadrado alcuadrado commented Jul 20, 2019

Maybe something like vyper-json --prepare-input file.vy | vyper-json?

The first call will take a file, and outputs a standarized json input with everything needed for compiling. That same json could be used for things like verifying the contract.

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Jul 21, 2019

Sure, as a shortcut tool. I was saying that vyper-json could work like so:

{
  "files": {
    "path/to/file1.vy": {
      "source": "# source code for file1..."
    },
    "path/to/file2.vy": {
      "source": "from path.to.file1 import interface..."
    }
  }
}

Internal to that script it would resolve the interfaces to the correct things.

@jacqueswww jacqueswww added the Approved label Jul 22, 2019
@alcuadrado

This comment has been minimized.

Copy link
Author

@alcuadrado alcuadrado commented Jul 23, 2019

Sure, as a shortcut tool.

Yes, sure. It should return the errors (e.g. parsing errors when extracting the imports) in the same format that vyper-json.

@iamdefinitelyahuman

This comment has been minimized.

Copy link
Contributor

@iamdefinitelyahuman iamdefinitelyahuman commented Jul 31, 2019

This functionality would be very useful to me, if nobody else is already working on it I'd be happy to start right away.

To minimize friction when integrating Vyper with existing tooling, I think it would be beneficial to format the output similarly to Solidity's standard JSON output. It would look something like this:

  {
    // Optional: not present if no errors/warnings were encountered
    "errors": [
      {
        "sourceLocation": {
          "file": "path/to/contract1.vy",
          "start": 1,
          "end": 20
        },
        "type": "TypeError",
        "severity": "error", // ("error" or "warning")
        "message": "A fatal error or warning"
      }
    ],
    "sources": {
      "path/to/contract1.vy": {
        "ast": {}
      }
    },
    "contracts": {
      "path/to/contract1.vy": {
        "ContractName": {
          "abi": [],
          "evm": {
            "bytecode": {
              "object": "",
              "opcodes": "",
              "sourceMap": ""
            },
            "deployedBytecode": { 
              "object": "",
              "opcodes": "",
              "sourceMap": ""
            },
            "methodIdentifiers": {}
          }
        }
      }
    }
  }

I'm also happy to follow the previously discussed format. However the above approach will make my life easier down the line, and likely that of others looking to integrate Vyper where Solidity is already in use. Any thoughts?

@alcuadrado

This comment has been minimized.

Copy link
Author

@alcuadrado alcuadrado commented Jul 31, 2019

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Jul 31, 2019

There are a number of things that should/could be standardized between the two compilers, and I think this would be good first steps towards making that happen.

@iamdefinitelyahuman

This comment has been minimized.

Copy link
Contributor

@iamdefinitelyahuman iamdefinitelyahuman commented Aug 1, 2019

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

Indeed, some of the objects will need to be handled in different ways depending on if they're coming from vyper or solc. But not all of them. A common format would allow a script that deploys bytecode, for example, to handle vyper or solc-based contracts via a single method. More complex use cases will need to differentiate, but they can still start from the same place and diverge on a per-object basis as needed, rather than starting from two places and sometimes converging.

I do think you raise a valid concern - we should not assume that the program processing the json is the same that generated it, so there needs to be an easy way to look at the data and know what to expect within the individual objects. What if we include a field such as "compiler": "vyper-v0.1.0.-beta.11"? That way we still have simplified integration, but can prevent the headaches caused by having to determine the compiler type through looking at how the source map is formatted.

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Aug 1, 2019

I believe compiler is a field generated by the output json anyways (in Solidity's output)

@charles-cooper

This comment has been minimized.

Copy link
Collaborator

@charles-cooper charles-cooper commented Aug 1, 2019

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

I don't believe standardization is within the scope of the VIP process but should be an EIP.

@fubuloubu

This comment has been minimized.

Copy link
Member

@fubuloubu fubuloubu commented Aug 1, 2019

Well yes, totally, I'm just saying starting down this road will eventually create the ability to standardize compiler interfaces for tools that use it, which would be best done in an EIP/ERC.

@jacqueswww

This comment has been minimized.

Copy link
Collaborator

@jacqueswww jacqueswww commented Aug 6, 2019

Note from meeting: make vyper-json as compliant to solc as possible.

@charles-cooper

This comment has been minimized.

@iamdefinitelyahuman

This comment has been minimized.

Copy link
Contributor

@iamdefinitelyahuman iamdefinitelyahuman commented Aug 6, 2019

Will start work on this shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.