Draft APIv2 request and responses

[kuzdogan]

See #1470

Currently in the process of drafting the API via Spotlight: https://sourcify.stoplight.io/docs/sourcify-apiv2/branches/main

Open questions:

• Should we have separate "single-file" and "multi-file" endpoints? IMO no, there no point in providing these APIs if the server will convert them to JSON anyway. The UI using this API can still provide a converter/wrapper for that functionality.
• In the /verify/metadata endpoint, should we have a separate "metadata" field (required) or just put everything in "files" field?
• In /verify/etherscan should we make the API key optional or required? Optional meaning by default it uses our own API key. Keep in mind we are easily reaching the rate limits of the API key.
• What HTTP code to return at the GET verify/{verificationId} endpoint if the verification failed. Still 200 or 500 etc?
• Biggest Q IMO: What fields to provide the user at the GET /contract/{chainId}/{address}` endpoint and in which format?

Action Items:

• Kaan to check how many of the Etherscan calls are done via our API vs custom keys.

[manuelwedler]

Some comments on this:

Should we have separate "single-file" and "multi-file" endpoints? IMO no, there no point in providing these APIs if the server will convert them to JSON anyway. The UI using this API can still provide a converter/wrapper for that functionality.

I agree here. The API should not support it, but the UI should have a converter to provide nice UX. If you are programmatically calling our API you are probably also able to create the standard json.

In the /verify/metadata endpoint, should we have a separate "metadata" field (required) or just put everything in "files" field?

In my opinion, to make it unambigous, we should add a "metadata" field. We would avoid errors in case of multiple metadata files and make it explicit that it is a required parameter.

Then, the question would be if we pass the metadata either as pure JSON, or as string encoded (which would be parsed by the server as it is now). I'm leaning more towards the second option as I see this endpoint mostly as a backward compatibility feature.

What HTTP code to return at the GET verify/{verificationId} endpoint if the verification failed. Still 200 or 500 etc?

500 only if the server experiences an error while handling the request. So it should be a 200 if the server is able to query the status of the provided verificationId. I would also not use the 202 here in case it is pending, and rather return 200 for all statuses. The reason is that you query the status of the job, not the contract as a resource.

Also, I would not return null in case the verification failed. Rather I would return "failed". I think we should avoid mixing types in the same field.

What fields to provide the user at the GET /contract/{chainId}/{address}` endpoint and in which format?

We can probably mostly orientate ourselves by the Blockscout API. The biggest question is here if we want to return the source code in the response, which will make the response payload quite big. However, in my opinion, it is easier to have just one endpoint that returns everything for a contract. We can still have a status endpoint that only returns the verification status of an address.

[kuzdogan]

Some discussion points during our call:

• How do we handle Hardhat output files?

  • Let's have a look at the usage metrics how many people use it.
  • We should keep the API lean and offload this parsing to the UI or a client.

• Should the /metadata endpoint accept an object or a string?

  • Object

  To add:

  • An endpoint to get the list of verified contracts on a chain, similar to /files/contracts/any/{chain}

[kuzdogan]

One important question is what to return in the /contract/{chainId}/{address} endpoint. This endpoint will be the main endpoint to check if a contract is verified on Sourcify, and also to retrieve detailed information about the contract.

The response by default will contain the minimal response:

{
  match: "exact_match" | "match" | null,
  creationMatch: "exact_match" | "match" | null,
  runtimeMatch: "exact_match" | "match" | null,
  chainId: "11155111",
  address: "0xfe34567890123456789012345678901234567890",
  verifiedAt: "2024-07-24T12:00:00Z"
}

In addition users can ask for other detailed fields in a query parameter e.g. /contracts/{chainId}/{address}?fields=abi,metadata,name or get everything but omit some: /contracts/{chainId}/{address}?omit=sources

We decided to define each field ourselves and not return a 1-to-1 reflection of the DB schema.

Now the tricky part is how to organize the bytecode information. We have the following data about the contract's bytecodes:

• onchain runtime bytecode
• onchain creation bytecode
• recompiled runtime bytecode
• recompiled creation bytecode
• transformations
  (i.e operations that need to be applied on top of the recompiled bytcodes to reach their onchain correspondent, explained here)
  keep in mind the "positions-" and the "values objects" of each of these have different fields in the DB schema
  • runtime transformations and their values:
    • cborAuxdata
    • library
    • immutable
    • callProtection
  • creation transformations and their values:
    • constructorArguments
    • cborAuxdata
    • library
• artifacts for the runtime bytecode
  • sourceMap
  • cborAuxdata
  • linkReferences
  • immutableReferences
• artifacts for the creation bytecode
  • sourceMap
  • cborAuxdata
  • linkReferences

The question is, how do we organize this data in an understandable way. Considering this available information here's my suggested output structure:

{
  // minimal info, returned always, can't be omitted
  match: "exact_match" | "match" | null,
  creationMatch: "exact_match" | "match" | null,
  runtimeMatch: "exact_match" | "match" | null,
  chainId: "11155111",
  address: "0xfe34567890123456789012345678901234567890",
  verifiedAt: "2024-07-24T12:00:00Z", 
  // bytecode details
  creationBytecode: { // not availible if creationMatch is `null`
    onchainBytecode: "0x608060405234801561001057...610150806100206000396000f3",
    recompiledBytecode: "0x608060405234801561001057...610150806100206000396000f3",
    // artifacts from the compiler output
    sourceMap: "368:7042:14:-:0;;;;;;;;;;;;;;;;;;;",
    linkReferences: {
      "/home/home/dotfiles/poof-core/contracts/MerkleTreeWithHistory.sol": {
        "Hasher": [
          {
            "start": 910,
            "length": 20
          },
        ],
      },
    },
    cborAuxdata: {
      "1": {
        "value": "0xa26469706673582212201e80049ede18eadf4ab7f0dea2c32f2375c33b5aef0b1a16cc5223dbc681559364736f6c63430007060033",
        "offset": 5471
      }
    },
    transformations: [
      {
        "id": "sources/lib/MyLib.sol:MyLib",
        "type": "replace",
        "offset": 582,
        "reason": "library"
      },
      {
        "id": "0",
        "type": "replace",
        "offset": 1269,
        "reason": "cborAuxdata"
      },
      {
        "type": "insert",
        "offset": 1322,
        "reason": "constructorArguments"
      }
    ],
    transformationValues: {
      "library": {
        "sources/lib/MyLib.sol:MyLib": "0x40b70a4904fad0ff86f8c901b231eac759a0ebb0"
      },
      "constructorArguments": "0x00000000000000000000000085fe79b998509b77bf10a8bd4001d58475d29386",
      "cborAuxdata": {
        "0": "0xa26469706673582212201c37bb166aa1bc4777a7471cda1bbba7ef75600cd859180fa30d503673b99f0264736f6c63430008190033"
      }
    }
  }, 
  runtimeBytecode: { // not availible if runtimeMatch is `null`
    onchainBytecode: "0x608060405234801561001057...610150806100206000396000f3",
    recompiledBytecode: "0x608060405234801561001057...610150806100206000396000f3",
    // artifacts from the compiler output
    sourceMap: "340:4705:14:-:0;;;;685:37..........59:1;4355;:5;;;;;;;4217:150;-1:-1:-1;;;4217:150:8:o;3136:155::-;3194:7;3226:1;3221;:6;;3213:49;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;-1:-1:-1;3279:5:8;;;3136:155::o",
    linkReferences: {
      "contracts/AmplificationUtils.sol": {
        "AmplificationUtils": [
          {
            "start": 3078,
            "length": 20
          },
        ],
      },
      "contracts/SwapUtils.sol": {
        "SwapUtils": [
          {
            "start": 2931,
            "length": 20
          },
        ],
      },
    },
    cborAuxdata: {
      "1": {
        "value": "0xa2646970667358221220fc5c3958995f3020259e8da7b7cfc439316c92fecdab62bef66bd9e7b1c0582864736f6c634300060c0033",
        "offset": 2026
      }
    },
    immutableReferences: {
      "1050": [
        {
          "start": 312,
          "length": 32
        },
        {
          "start": 2631,
          "length": 32
        }
      ]
    };
    // transformations applied to the recompiled bytecode to reach the onchain bytecode
    transformations: [
      {
        "id": "20",
        "type": "replace",
        "offset": 137, // in bytes
        "reason": "immutable"
      },
      {
        "id": "0",
        "type": "replace",
        "offset": 1002,
        "reason": "auxdata"
      }
    ];
    transformationValues: {
      "immutables": {
        "20": "0x000000000000000000000000b5d83c2436ad54046d57cd48c00d619d702f3814"
      },
      "cborAuxdata": {
        "0": "0xa26469706673582212205817b060c918294cc8107069c4fd0a74dbfc35b0617214043887fe9d4e17a4a864736f6c634300081a0033"
      }
    }
  },
  deployment: {
    transactionHash: "0xa0483d0b361b247883268883842e66b248952148898849884884884884884884",
    blockNumber: 35453621;
    transactionIndex: 2,
    deployer: "0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984",
  },
  compilation: {
    sources: {
      "contracts/Storage.sol": { 
        id: 0 // The AST identifiers of sources
        content: "// SPDX-License-Identifier: MIT\npragma solidity ^0.8.0;\n\ncontract Storage {\n    uint256 number;\n\n    function setNumber(uint256 newNumber) public {\n        number = newNumber;\n    }\n\n    function getNumber() public view returns (uint256) {\n        return number;\n    }\n}\n",
      },
      "contracts/Owner.sol": {
        id: 1
        content: "// SPDX-License-Identifier: MIT\npragma solidity ^0.8.0;\n\ncontract Owner {\n    address public owner;\n\n    constructor() {\n        owner = msg.sender;\n    }\n}\n"
      }
    },
    language: "Solidity",
    compilerVersion: "v0.8.12+commit.f00d7308",
    compilerSettings: {}
    name: "Storage",
    fullyQualifiedName: "contracts/Storage.sol:Storage",
  }
  abi: [],
  userDoc: {},
  devDoc: {},
  storageLayout: {},
  metadata: {},
}

[kuzdogan]

Please do a final short review on the Draft here: https://sourcify.stoplight.io/docs/sourcify-apiv2/branches/main

@marcocastignoli @manuelwedler

[marcocastignoli]

1. The verificationJob object will not last forever, we have to clean the completed ones. Would it make sense to also include the "expiration date" of a verificationJob?
2. proxy support is missing Handling "Minimal Proxy Contract" pattern #1624, I think we should include it
3. should we include also sources.hashes and ipfs/swarm links in the /v2/contract/chainId/address response?
4. minor: I see custom_code is used everywhere with the same example "unsupported_chain", but I think we can create two different custom_code: one for /v2/verify/chainId/address fail statuses and another for /v2/verify/verificationId

[marcocastignoli]

Also another big doubt: will we support RepositoryVXStorageService as valid read services with apiV2? If that's the case then many fields of the /v2/contract/chainId/address response will not be available