Barge issues: pdr-trueval, connecting subgraph #194

trentmc · 2023-10-04T12:07:18Z

See attached log.
temp.txt

trentmc · 2023-10-04T14:05:36Z

One more error:

ocean-ocean-contracts-1  | [Error: EBUSY: resource busy or locked, rmdir '/ocean-contracts/artifacts'] {
ocean-ocean-contracts-1  |   errno: -16,
ocean-ocean-contracts-1  |   code: 'EBUSY',
ocean-ocean-contracts-1  |   syscall: 'rmdir',
ocean-ocean-contracts-1  |   path: '/ocean-contracts/artifacts'
ocean-ocean-contracts-1  | }
ocean-ocean-contracts-1  | Warning: Visibility for constructor is ignored. If you want the contract to be non-deployable, making it "abstract" is sufficient.
ocean-ocean-contracts-1  |   --> contracts/utils/OceanToken.sol:33:5:
ocean-ocean-contracts-1  |    |
ocean-ocean-contracts-1  | 33 |     constructor(
ocean-ocean-contracts-1  |    |     ^ (Relevant source part starts here and spans across multiple lines).

alexcos20 · 2023-10-04T14:06:19Z

One more error:

ocean-ocean-contracts-1  | [Error: EBUSY: resource busy or locked, rmdir '/ocean-contracts/artifacts'] {
ocean-ocean-contracts-1  |   errno: -16,
ocean-ocean-contracts-1  |   code: 'EBUSY',
ocean-ocean-contracts-1  |   syscall: 'rmdir',
ocean-ocean-contracts-1  |   path: '/ocean-contracts/artifacts'
ocean-ocean-contracts-1  | }
ocean-ocean-contracts-1  | Warning: Visibility for constructor is ignored. If you want the contract to be non-deployable, making it "abstract" is sufficient.
ocean-ocean-contracts-1  |   --> contracts/utils/OceanToken.sol:33:5:
ocean-ocean-contracts-1  |    |
ocean-ocean-contracts-1  | 33 |     constructor(
ocean-ocean-contracts-1  |    |     ^ (Relevant source part starts here and spans across multiple lines).

that is just a warning, you can ignore it

trentmc · 2023-10-04T14:07:01Z

address.json.txt

alexcos20 · 2023-10-04T14:07:46Z

See attached log. temp.txt

Looks like pdr-backend is trying to connect to "172.15.0.15:8000". But on MacOs, container do not have ip addresses, so we need to find another way to connect to subgraph (maybe something like host.docker.internal:8000 ?). I don't use a Mac, so I cannot test it

alexcos20 · 2023-10-04T14:08:21Z

address.json.txt

all good, all contracts are deployed on "development" network

trizin · 2023-10-04T16:18:10Z

I don't see any log messages from the subgraph, could it be down? Could you check the subgraph logs?

trentmc · 2023-10-06T09:46:42Z

I rebooted everything, cleaned everything, it's working now

My guess it was a subgraph issue. But now it's all good.

Thanks everyone for your help.

trentmc · 2023-10-06T11:39:11Z

So I had it going, good.

Then I stopped it, and re-started it. Now it's failing again :(. Looks like a subgraph error. Here's the log from the subgraph docker process:

...
- Generate types for GraphQL schema
✔ Generate types for GraphQL schema

Types generated successfully


> ocean-subgraph@3.0.9 create:local-barge
> graph create oceanprotocol/ocean-subgraph --node http://172.15.0.15:8020

- Creating subgraph in Graph node: http://172.15.0.15:8020/
✖ HTTP error creating the subgraph: EHOSTUNREACH
/usr/src/app/node_modules/@oclif/core/lib/errors/index.js:21
    throw new exit_2.ExitError(code);
    ^

ExitError: EEXIT: 1
    at Object.exit (/usr/src/app/node_modules/@oclif/core/lib/errors/index.js:21:11)
    at CreateCommand.exit (/usr/src/app/node_modules/@oclif/core/lib/command.js:131:23)
    at /usr/src/app/node_modules/@graphprotocol/graph-cli/dist/commands/create.js:46:22
    at ClientHttp.Client._parseResponse (/usr/src/app/node_modules/jayson/lib/client/index.js:190:12)
    at /usr/src/app/node_modules/jayson/lib/client/index.js:149:10
    at ClientRequest.<anonymous> (/usr/src/app/node_modules/jayson/lib/client/http.js:100:7)
    at ClientRequest.emit (node:events:513:28)
    at Socket.socketErrorListener (node:_http_client:494:9)
    at Socket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:157:8) {
  oclif: { exit: 1 },
  code: 'EEXIT'
}

trizin · 2023-10-06T11:41:41Z

Looks like the subgraph is unavailable, could you check if it's running and inspect the logs if it's running?

trentmc · 2023-10-06T11:44:12Z

Yes, exactly, the subgraph is unavailable.

That error listed above was from the docker subgraph process.

Here's the "liveness" of the process:

trentmc@tlm-macbook: ~/code/barge $ docker ps -a|grep ubgr
94d6837431e7   oceanprotocol/subgraph:predictoor          "/usr/src/app/docker…"   7 hours ago   Up 7 minutes                                                                                         ocean-subgraph-1

And, from trentmc@tlm-macbook: ~ $ docker logs 94d6837431e7:

here's the full log: log.txt
the error reported in the comment above is at the bottom of the log

trizin · 2023-10-06T12:07:30Z

Thanks for providing the logs. Looks interesting, I haven't seen something like this before. Could be related to what @alexcos20 mentioned above #194 (comment) although it's weird that it works from time to time and then fails

trentmc · 2023-10-06T12:26:28Z

[Berkay] Could be related to what @alexcos20 mentioned

[what @alexcos20 mentioned] But on MacOs, container do not have ip addresses, so we need to find another way to connect to subgraph (maybe something like host.docker.internal:8000 ?)

Yet this was working two weeks ago. And earlier today, if briefly.

Perhaps it's the following. There are two possible subgraph URLs:

export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"
export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"

IIRC one was working (top one?) and the other wasn't.

trizin · 2023-10-06T12:52:40Z

Yep first one should work. However this error occurs in the subgraph container while deploying the subgraph. Perhaps try a clean start by running the cleanup script and removing ~/.ocean folder?

trentmc · 2023-10-06T13:46:30Z

Perhaps try a clean start by running the cleanup script and removing ~/.ocean folder?

Yep, WIP.

I hit another snag: earlier today "to be prudent" I upgraded my Docker from 4.22 to 4.24. Then Docker kept randomly hanging. After googling around, it seems many people had that issue, and the fix was to go back to 4.22.

I'll keep you posted.

trentmc · 2023-10-06T13:54:06Z

Docker seems to be behaving better now.

However, the oceanprotocol/subgraph:predictoor container has this as its full log. Something's up I think?

trentmc@tlm-macbook: ~ $ docker logs -f 204871861e18
deploy subgraph is true
Waiting for contracts to be deployed

> ocean-subgraph@3.0.9 quickstart:barge
> node ./scripts/generatenetworkssubgraphs.js development && npm run codegen && npm run create:local-barge && npm run deploy:local-barge

Using custom ADDRESS_FILE instead of ocean-contracts npm dep
Skipping mumbai
Skipping polygon
...
Skipping sepolia
Skipping oasis_saphire

> ocean-subgraph@3.0.9 codegen
> graph codegen --output-dir src/@types

    Error: ENOENT: no such file or directory, open 'subgraph.yaml'
    Code: ENOENT

trentmc · 2023-10-06T14:25:27Z

Good news! I think in the previous run, while I'd run the cleanup script, I had missed deleting ~/.ocean. Because in my next run, I did do that and I think it's working.

Some logging from subgraph container is below. Fingers crossed...

...
Skipping oasis_saphire
Creating subgraph.yaml for development
	 Adding veOCEAN

> ocean-subgraph@3.0.9 codegen
> graph codegen --output-dir src/@types

- Apply migrations
...
- Generate types for GraphQL schema
✔ Generate types for GraphQL schema

Types generated successfully


> ocean-subgraph@3.0.9 create:local-barge
> graph create oceanprotocol/ocean-subgraph --node http://172.15.0.15:8020

- Creating subgraph in Graph node: http://172.15.0.15:8020/
Created subgraph: oceanprotocol/ocean-subgraph

> ocean-subgraph@3.0.9 deploy:local-barge
> graph deploy oceanprotocol/ocean-subgraph subgraph.yaml -l $npm_package_version --ipfs http://172.15.0.16:5001 --node http://172.15.0.15:8020
...
  Compile data source: veFeeDistributor => build/veFeeDistributor/veFeeDistributor.wasm
- Compile subgraph
...
- Upload subgraph to IPFS
                .. QmVHM3xfMTNQUrC3psri54SeFMpQZSEWMH1X7tdM6CtMXX
- Upload subgraph to IPFS
✔ Upload subgraph to IPFS

Build completed: QmQfEowKwF6kJUXRGrjx8usvrpV8hw3nyW57QTAv5HHxF4

- Deploying to Graph node http://172.15.0.15:8020/
Deployed to http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph/graphql

Subgraph endpoints:
Queries (HTTP):     http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph

Summary of lessons so far:

In MacOS, use Docker 4.22 not 4.24 which freezes.
Before running barge, do all three cleanups, as follows. (I've already updated barge.md accordingly)
```
rm -rf ~/.ocean
./cleanup.sh
docker system prune -a --volumes
```

trizin · 2023-10-06T14:31:46Z

Great!

trentmc · 2023-10-06T14:34:30Z

Alas, I am still having trouble getting predictoor to see the subgraph.

In console, I do the following:

cd code/pdr-backend/

# Setup virtualenv
cd pdr-backend
source venv/bin/activate

# Set envvars
export ADDRESS_FILE="${HOME}/.ocean/ocean-contracts/artifacts/address.json"
export RPC_URL=http://127.0.0.1:8545
export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"
export PRIVATE_KEY="0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58"

# run random predictoor bot (agent)
python pdr_backend/predictoor/main.py 1

And the result I get from the last line is:

HTTPConnectionPool(host='172.15.0.15', port=8000): Max retries exceeded with url: /subgraphs/name/oceanprotocol/ocean-subgraph (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x1081a6690>, 'Connection to 172.15.0.15 timed out. (connect timeout=1.5)'))
No feeds found. Exiting

Also, the subgraph url of interest does not load in my browser. http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph

Help? @trizin

trizin · 2023-10-06T14:35:20Z

Try to use the localhost:9000 for subgraph url

trentmc · 2023-10-06T14:37:11Z

Try to use the localhost:9000 for subgraph url

Aha! And that works! My predictoor log:

(venv) trentmc@tlm-macbook: export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"

(venv) trentmc@tlm-macbook: ~/code/pdr-backend $ python pdr_backend/predictoor/main.py 1

--------------------------------------------------------------------------------
Config:
PredictoorConfig1={private_key=0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58, rpc_url=http://127.0.0.1:8545, s_until_epoch_end=60, stake_amount=1, subgraph_url=http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph, _abc_impl=<_abc._abc_data object at 0x104c9ba40>, owner_addresses=[], pair_filters=[], source_filter=[], timeframe_filter=[], web3_config=<pdr_backend.util.web3_config.Web3Config object at 0x101bf3f50> /PredictoorConfig1}

................................................................................
Feeds (detailed):
...

Starting main loop...

--------------------------------------------------------------------------------
Take_step() begin.
  block_number=793, prev=0
  Got new block. Timestamp={block['timestamp']}
    Process [Feed 0x0356a ETH/USDT|binance|5m] at epoch=5655343
      282 s left in epoch (predict if <= 60 s left)
      Done feed: too early to predict
    Process [Feed 0x4bde0 BTC/TUSD|binance|5m] at epoch=5655343
...

And that url (http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph) also works in my browser, bringing up the expected text boxes for entering queries.

trentmc · 2023-10-06T14:37:37Z

This issue is solved, hooray:)

trentmc · 2023-10-06T14:41:38Z

@trizin so that I can update my own mental model:

The subgraph container reports a subgraph url of http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph
Yet predictoor agent & browser fail on that url
The agent & browser do work for this url: http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph

What's the relation between these urls? Why is it a remote IP address for the docker container, and a local (IP address?) for the agent & browser? Thanks!

alexcos20 · 2023-10-06T14:44:42Z

because docker on macOS does not support brige, which is the driver that we are using to create and assign ip addrs to all containers (https://docker-docs.uclv.cu/docker-for-mac/networking/#there-is-no-docker0-bridge-on-macos)

alexcos20 · 2023-10-06T14:45:06Z

and https://docker-docs.uclv.cu/docker-for-mac/networking/#per-container-ip-addressing-is-not-possible

trentmc · 2023-10-06T15:19:20Z

Thanks Alex!

trizin · 2023-10-06T15:21:02Z

According to my knowledge:

https://github.com/oceanprotocol/barge/blob/main/compose-files/thegraph.yml#L6 This exposes the port 8000 in the container to the host network, thus it's accessible.

"172.15.0.15" is the internal ip of the container, other containers in the network can access it. As far as I know the docker network is isolated from host network unless you expose a port so you cannot access it.

trentmc added the Type: Bug Something isn't working label Oct 4, 2023

trentmc closed this as completed Oct 6, 2023

trentmc reopened this Oct 6, 2023

trentmc added a commit that referenced this issue Oct 6, 2023

For #194 and generally: add more cleanup info to barge.md

772dcd9

trentmc mentioned this issue Oct 6, 2023

For #194 and generally: add more cleanup info to barge.md #203

Merged

trentmc closed this as completed in #203 Oct 6, 2023

trentmc added a commit that referenced this issue Oct 6, 2023

For #194 and generally: add more cleanup info to barge.md (#203)

6aebb82

trentmc reopened this Oct 6, 2023

trentmc closed this as completed Oct 6, 2023

This was referenced Oct 7, 2023

MacOS M2 freezes on Docker 4.24 but not 4.22. Document it in barge.md #206

Closed

Clarify subgraph url, to avoid MacOS issues #209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Barge issues: pdr-trueval, connecting subgraph #194

Barge issues: pdr-trueval, connecting subgraph #194

trentmc commented Oct 4, 2023

trentmc commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

trentmc commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

trizin commented Oct 4, 2023

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 •

edited

Loading

trizin commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 •

edited

Loading

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023

alexcos20 commented Oct 6, 2023

alexcos20 commented Oct 6, 2023

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023 •

edited

Loading

Barge issues: pdr-trueval, connecting subgraph #194

Barge issues: pdr-trueval, connecting subgraph #194

Comments

trentmc commented Oct 4, 2023

trentmc commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

trentmc commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

alexcos20 commented Oct 4, 2023

trizin commented Oct 4, 2023

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 • edited Loading

trizin commented Oct 6, 2023 • edited Loading

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023 • edited Loading

trentmc commented Oct 6, 2023 • edited Loading

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 • edited Loading

trizin commented Oct 6, 2023

trentmc commented Oct 6, 2023 • edited Loading

trentmc commented Oct 6, 2023

trentmc commented Oct 6, 2023

alexcos20 commented Oct 6, 2023

alexcos20 commented Oct 6, 2023

trentmc commented Oct 6, 2023

trizin commented Oct 6, 2023 • edited Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trizin commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trentmc commented Oct 6, 2023 •

edited

Loading

trizin commented Oct 6, 2023 •

edited

Loading