Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

producer and indexer error #349

Closed
liuxh-go opened this issue Mar 1, 2021 · 1 comment
Closed

producer and indexer error #349

liuxh-go opened this issue Mar 1, 2021 · 1 comment

Comments

@liuxh-go
Copy link

liuxh-go commented Mar 1, 2021

Producer operation error, the log is as follows:

(docker logs -f ava_producer_1)

INFO [03-01|02:17:32] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#319: starting processing block -1
[2021-03-01T02:17:32.982266296Z]: job:update-tx-pool event:dbr.exec.exec err:Error 1146: Table 'ortelius.tx_pool' doesn't exist kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2PnhAqiebtfg8uQ4gV7R8jeptxS7YgreiTbTrUJRn22yz7gLhs',1,'2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5','2pAo2eMuKaMyhKWAyyKZe6FKwfEnDczfYjgYcLbJcJKzrLGb6F',?,0,'1-2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5-cchain','2021-03-01 02:17:32.925920')]
[2021-03-01T02:17:32.982280517Z]: job:update-tx-pool event:dbr.exec time:607 μs kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2PnhAqiebtfg8uQ4gV7R8jeptxS7YgreiTbTrUJRn22yz7gLhs',1,'2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5','2pAo2eMuKaMyhKWAyyKZe6FKwfEnDczfYjgYcLbJcJKzrLGb6F',?,0,'1-2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5-cchain','2021-03-01 02:17:32.925920')]
ERROR[03-01|02:17:32] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#475: Unknown error: Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool)
INFO [03-01|02:17:32] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#393: close producer 1 2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5 cchain
[2021-03-01T02:17:32.982322394Z]: job:general event:close
INFO [03-01|02:17:32] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#509: Exiting worker for cchain
ERROR[03-01|02:17:32] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#340: Error running worker: Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool)
[2021-03-01T02:17:33.00088649Z]: job:write-buffer event:dbr.exec.exec err:Error 1146: Table 'ortelius.tx_pool' doesn't exist kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('ZJyGCcpBWXgaFgib1mFEaoQq86eVdV7GpBgnTbYSwSk3VLcib',1,'11111111111111111111111111111111LpoYY','2YHbbBzAtykroNag7jKToe5XZsnRHifJsNtfiMYJjgnLANXzpL',?,0,'1-11111111111111111111111111111111LpoYY-decisions','2021-03-01 02:17:28.997657')]
[2021-03-01T02:17:33.000899854Z]: job:write-buffer event:dbr.exec time:429 μs kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('ZJyGCcpBWXgaFgib1mFEaoQq86eVdV7GpBgnTbYSwSk3VLcib',1,'11111111111111111111111111111111LpoYY','2YHbbBzAtykroNag7jKToe5XZsnRHifJsNtfiMYJjgnLANXzpL',?,0,'1-11111111111111111111111111111111LpoYY-decisions','2021-03-01 02:17:28.997657')]
WARN [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/write_buffer.go#256: Error writing to db (retry):%!(EXTRA *fmt.wrapError=Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool))
[2021-03-01T02:17:33.001367943Z]: job:write-buffer event:dbr.exec.exec err:Error 1146: Table 'ortelius.tx_pool' doesn't exist kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2Cc9zanTncTyeenYHF7fXj1b4gm5QgUkdfy3aVHxmN6eMToKrm',1,'11111111111111111111111111111111LpoYY','2YHbbBzAtykroNag7jKToe5XZsnRHifJsNtfiMYJjgnLANXzpL',?,0,'1-11111111111111111111111111111111LpoYY-consensus','2021-03-01 02:17:27.997342')]
[2021-03-01T02:17:33.001379821Z]: job:write-buffer event:dbr.exec time:399 μs kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2Cc9zanTncTyeenYHF7fXj1b4gm5QgUkdfy3aVHxmN6eMToKrm',1,'11111111111111111111111111111111LpoYY','2YHbbBzAtykroNag7jKToe5XZsnRHifJsNtfiMYJjgnLANXzpL',?,0,'1-11111111111111111111111111111111LpoYY-consensus','2021-03-01 02:17:27.997342')]
WARN [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/write_buffer.go#256: Error writing to db (retry):%!(EXTRA *fmt.wrapError=Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool))
INFO [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#412: Starting worker for cchain
[2021-03-01T02:17:33.183998403Z]: job:get-block event:dbr.select time:298 μs kvs:[sql:SELECT cast(case when max(block) is null then -1 else max(block) end as char) as block FROM cvm_blocks]
INFO [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#319: starting processing block -1
[2021-03-01T02:17:33.246287019Z]: job:update-tx-pool event:dbr.exec.exec err:Error 1146: Table 'ortelius.tx_pool' doesn't exist kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2PnhAqiebtfg8uQ4gV7R8jeptxS7YgreiTbTrUJRn22yz7gLhs',1,'2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5','2pAo2eMuKaMyhKWAyyKZe6FKwfEnDczfYjgYcLbJcJKzrLGb6F',?,0,'1-2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5-cchain','2021-03-01 02:17:33.187759')]
[2021-03-01T02:17:33.24630214Z]: job:update-tx-pool event:dbr.exec time:502 μs kvs:[sql:INSERT INTO `tx_pool` (`id`,`network_id`,`chain_id`,`msg_key`,`serialization`,`processed`,`topic`,`created_at`) VALUES ('2PnhAqiebtfg8uQ4gV7R8jeptxS7YgreiTbTrUJRn22yz7gLhs',1,'2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5','2pAo2eMuKaMyhKWAyyKZe6FKwfEnDczfYjgYcLbJcJKzrLGb6F',?,0,'1-2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5-cchain','2021-03-01 02:17:33.187759')]
ERROR[03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#475: Unknown error: Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool)
INFO [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#393: close producer 1 2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5 cchain
[2021-03-01T02:17:33.246354515Z]: job:general event:close
INFO [03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#509: Exiting worker for cchain
ERROR[03-01|02:17:33] /go/src/github.com/ava-labs/ortelius/stream/producer_cchain.go#340: Error running worker: Error 1146: Table 'ortelius.tx_pool' doesn't exist (tx_pool)

Indexer operation error, the log is as follows

(docker logs -f ava_indexer_1)

2021/03/01 02:20:08 Failed to run: Failed to index avm genesis tx 0
 --- at /go/src/github.com/ava-labs/ortelius/services/indexes/avm/writer.go:253 (Writer.insertGenesis) ---
Caused by: Error 1054: Unknown column 'transaction_id' in 'field list' (output_addresses_accumulate_out)
INFO [03-01|02:21:10] /go/src/github.com/ava-labs/ortelius/stream/consumers/indexer.go#100: bootstrap 1 vm pvm chain 11111111111111111111111111111111LpoYY
INFO [03-01|02:21:11] /go/src/github.com/ava-labs/ortelius/stream/consumers/indexer.go#108: bootstrap complete 1 vm avm chain 2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM
INFO [03-01|02:21:11] /go/src/github.com/ava-labs/ortelius/stream/consumers/indexer.go#100: bootstrap 1 vm avm chain 2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM
INFO [03-01|02:21:12] /go/src/github.com/ava-labs/ortelius/stream/consumers/indexer.go#108: bootstrap complete 1 vm avm chain 2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM
[2021-03-01T02:21:12.684095903Z]: job:general event:close
2021/03/01 02:21:12 Failed to run: Failed to index avm genesis tx 0
 --- at /go/src/github.com/ava-labs/ortelius/services/indexes/avm/writer.go:253 (Writer.insertGenesis) ---
Caused by: Error 1054: Unknown column 'transaction_id' in 'field list' (output_addresses_accumulate_out)

docker ps view and print as follows

CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS                          PORTS                                              NAMES
054b1d9f196c        avaplatform/avalanchego:v1.2.0          "/bin/sh -cx 'exec .…"   44 minutes ago      Up 44 minutes                   127.0.0.1:9650->9650/tcp                           ava_avalanche_1
ee66b7f97d1d        avaplatform/ortelius:c562648            "/opt/orteliusd api …"   44 minutes ago      Up 43 minutes                   127.0.0.1:8080->8080/tcp                           ava_api_1
893335b0da78        avaplatform/ortelius:c562648            "/opt/orteliusd stre…"   44 minutes ago      Restarting (1) 40 seconds ago                                                      ava_indexer_1
77b52de7d235        avaplatform/ortelius:c562648            "/opt/orteliusd stre…"   44 minutes ago      Up 44 minutes                                                                      ava_producer_1
45203d1bd38e        confluentinc/cp-kafka:5.4.3             "/etc/confluent/dock…"   44 minutes ago      Up 44 minutes                   0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   ava_kafka_1
973f0b9b1c39        mysql:8.0.22                            "docker-entrypoint.s…"   44 minutes ago      Up 44 minutes                   0.0.0.0:3306->3306/tcp, 33060/tcp                  ava_mysql_1
c724993dc3b6        redis:6.0.9-alpine3.12                  "docker-entrypoint.s…"   44 minutes ago      Up 44 minutes                   0.0.0.0:6379->6379/tcp                             ava_redis_1
0a3d1e1f44d8        confluentinc/cp-zookeeper:5.4.3         "/etc/confluent/dock…"   44 minutes ago      Up 44 minutes                   2888/tcp, 3888/tcp, 0.0.0.0:32774->2181/tcp        ava_zookeeper_1

This is my docker-compose.yml file

version: '3.5'
volumes:
  avalanche-ipcs:
services:
  mysql:
    image: "mysql:8.0.22"
    volumes:
      - ./data/mysql:/var/lib/mysql
      - ./my.cnf:/etc/mysql/my.cnf
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: ortelius
    restart: on-failure
  migrate:
    image: "migrate/migrate:v4.13.0"
    volumes:
      - ./ortelius/services/db/migrations:/migrations
    depends_on:
      - mysql
    entrypoint: ["/bin/sh"]
    command: |
      -c 'while ! migrate -path=/migrations/ -database "mysql://root:password@tcp(mysql:3306)/ortelius" up; do
        sleep 1
      done'
    restart: on-failure
  redis:
    image: "redis:6.0.9-alpine3.12"
    command: redis-server
    ports:
      - "6379:6379"
    restart: on-failure
  zookeeper:
    image: "confluentinc/cp-zookeeper:5.4.3"
    ports:
      - 2181
    environment:
      - ZOOKEEPER_SERVER_ID=1
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_SERVERS=zookeeper:4182:5181
    volumes:
      - ./data/zookeeper/data:/var/lib/zookeeper/data/
      - ./log/zookeeper/logs:/var/lib/zookeeper/log/
    restart: on-failure
  kafka:
    image: "confluentinc/cp-kafka:5.4.3"
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "29092:29092"
    volumes:
      - ./data/kafka:/var/lib/kafka/data/
    restart: on-failure
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29092

      KAFKA_BROKER_ID: 1
      CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      KAFKA_HEAP_OPTS: -Xms256M -Xmx256M -verbose:gc
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_NUM_PARTITIONS: 8
      KAFKA_OFFSETS_RETENTION_MINUTES: 446400

      # Disable replication and lower thread count
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_MIN_INSYNC_REPLICAS: 1

      KAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR: 1
      KAFKA_NUM_NETWORK_THREADS: 3
      KAFKA_NUM_IO_THREADS: 3

      # Set retention policies
      KAFKA_LOG_CLEANUP_POLICY: compact
      KAFKA_LOG_RETENTION_BYTES: -1
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
      KAFKA_LOG_RETENTION_HOURS: -1
      KAFKA_LOG_ROLL_HOURS: 24
      KAFKA_LOG_SEGMENT_BYTES: 1048576
      KAFKA_LOG_SEGMENT_DELETE_DELAY_MS: 60000
  avalanche:
    env_file:
      - ./production.env
    image: "avaplatform/avalanchego:v1.2.0"
    command: /bin/sh -cx "exec ./build/avalanchego
      --network-id=$${NETWORKID}
      --db-dir=/var/lib/avalanche
      --log-level=info
      --http-host=0.0.0.0
      --ipcs-chain-ids=$${P_CHAINID},$${X_CHAINID}
      --coreth-config='{\"rpc-gas-cap\":2500000000,\"rpc-tx-fee-cap\":100,\"eth-api-enabled\":true,\"debug-api-enabled\":true,\"tx-pool-api-enabled\":true}'
      "
    ports:
      - 127.0.0.1:9650:9650
    volumes:
      - ./data/avalanche:/var/lib/avalanche
      - avalanche-ipcs:/tmp
    depends_on:
      - producer
    restart: always
  kafkatopics:
    env_file:
      - ./production.env
    depends_on:
      - kafka
    image: "confluentinc/cp-kafka:5.4.3"
    command: bash -cx "kafka-topics --bootstrap-server $${KAFKA_HOST} --list &&
      kafka-topics --create --if-not-exists --zookeeper $${ZOOKEEPER_HOST} --replication-factor $${KAFKA_REPLICATIONFACTOR} --partitions $${KAFKA_NUMPARTITIONS} --topic $${NETWORKID}-$${C_CHAINID}-cchain &&
      kafka-topics --create --if-not-exists --zookeeper $${ZOOKEEPER_HOST} --replication-factor $${KAFKA_REPLICATIONFACTOR} --partitions $${KAFKA_NUMPARTITIONS} --topic $${NETWORKID}-$${P_CHAINID}-consensus &&
      kafka-topics --create --if-not-exists --zookeeper $${ZOOKEEPER_HOST} --replication-factor $${KAFKA_REPLICATIONFACTOR} --partitions $${KAFKA_NUMPARTITIONS} --topic $${NETWORKID}-$${P_CHAINID}-decisions &&
      kafka-topics --create --if-not-exists --zookeeper $${ZOOKEEPER_HOST} --replication-factor $${KAFKA_REPLICATIONFACTOR} --partitions $${KAFKA_NUMPARTITIONS} --topic $${NETWORKID}-$${X_CHAINID}-consensus &&
      kafka-topics --create --if-not-exists --zookeeper $${ZOOKEEPER_HOST} --replication-factor $${KAFKA_REPLICATIONFACTOR} --partitions $${KAFKA_NUMPARTITIONS} --topic $${NETWORKID}-$${X_CHAINID}-decisions
      "
  producer: &ortelius-app
    image: "avaplatform/ortelius:c562648"
    command: ["stream", "producer", "-c", "/opt/config.json"]
    external_links:
      - zookeeper
      - kafka
      - mysql
      - redis
    depends_on:
      - kafkatopics
    volumes:
      - avalanche-ipcs:/tmp
    restart: on-failure
  indexer:
    <<: *ortelius-app
    command: ["stream", "indexer", "-c", "/opt/config.json"]
  api:
    <<: *ortelius-app
    command: ["api", "-c", "/opt/config.json"]
    ports:
      - 127.0.0.1:8080:8080

How should I solve this problem?
@tasinco
Copy link
Contributor

tasinco commented Mar 1, 2021

Error 1146: Table 'ortelius.tx_pool' doesn't exist

https://github.com/ava-labs/ortelius/blob/master/docs/production.md

It seems not all your migrations ran. Necessary db updates https://github.com/ava-labs/ortelius/tree/master/services/db/migrations

Make sure that all updates have completed in the order of creation.

@tasinco tasinco closed this as completed Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@liuxh-go @tasinco