refactor!: use chainhook to listen for chain events instead of a direct stacks api connection #200

rafaelcr · 2024-03-03T17:35:12Z

This PR replaces the direct connection between this API and the Stacks Blockchain API DB with a Chainhook predicate that feeds every Stacks block for analysis. This API then checks for token contract deployments, metadata notifications, and token mints in order to fetch their metadata.

Add re-org support
Group tests in suites for parallel GH action runs
Migrate to node v20
Persist chainhook predicates in local disk for retrieval upon restarts
Rewrite image cache processing in TypeScript and move to a step-by-step image transformation where local files are used temporarily to optimize memory use
Improve error reporting so if tokens have user-faults (metadata not parseable, unreachable URLs, size too large, etc.) we print the correct error in token responses
Add profiler server
Include admin RPCs to start/stop job queue at will

This PR significantly changes the DB schema so it represents a breaking change.

github-actions · 2024-03-03T17:36:11Z

Vercel deployment URL: https://token-metadata-eco3k2mc9-hirosystems.vercel.app 🚀

* fix: image cache ts * test: more * fix: code * build: use v2 docker compose * fix: admin tests * fix: try finally * test: debug logs * test: only 1 * test: one more * fix: not the timeout * fix: different ports * fix: improve test servers * fix: add cause to error message * fix: try 127.0.0.1 * fix: try timeout sleep * fix: standard close * test: more * fix: improve message * fix: echo status * fix: detect * fix: finally * fix: x * fix: timeouts * fix: revert debug stuff * chore: remove old test

…233) fix: add tests

* fix: try to use native sharp pipeline * fix: job queue controls, profiler * fix: tests * fix: upgrade toolkit * fix: improve error propagation * fix: listen to stream errors * fix: process cached image step by step * fix: report errors retried immediately * fix: image stream * fix: wait for sharp stream correctly

…errors (#236) * chore: demote httperror to usererror * fix: add errors * fix: custom messages

zone117x · 2024-08-21T11:11:34Z

src/token-processor/util/sip-validation.ts

@@ -361,17 +370,25 @@ export function getContractLogMetadataUpdateNotification(
  }
 }

+export type NftMintEvent = {
+  contractId: string;
+  tokenId: bigint;


There are a few areas where token-id int sizes are different, like in some postgres tables it's an int32. Anything to worry about?

Good catch, I'll make a pass on these to double check

src/chainhook/server.ts

zone117x · 2024-08-21T11:20:06Z

src/token-processor/images/image-cache.ts

+  }
+}
+
+async function uploadImage(localPath: string, remoteName: string): Promise<string> {


Could be a good time to switch to @google-cloud/storage #215 it would simplify a lot of the auth and streaming here. Just a suggestion, this code seems fine too

Yeah for sure... I'll prioritize that issue to work on it next

zone117x · 2024-08-21T11:21:21Z

src/pg/types.ts

@@ -86,6 +109,17 @@ export type DbJob = {
  updated_at?: string;
 };

+export type DbUpdateNotification = {
+  token_id: number;


Token-id is type number here, just a heads up in case it needs to handle larger int sizes

Ah, that particular token ID is a postgres ID, not a bigint clarity token ID

zone117x

Awesome refactor! Left some nits but overall looks great 💯

## [1.0.0-beta.1](v0.7.0...v1.0.0-beta.1) (2024-08-21) ### ⚠ BREAKING CHANGES * use chainhook to listen for chain events instead of a direct stacks api connection (#200) ### Bug Fixes * use prometheus port configured in ENV ([c769d29](c769d29)) ### Code Refactoring * use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)

blockstack-devops · 2024-08-21T16:09:12Z

🎉 This PR is included in version 1.0.0-beta.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

## [1.0.0](v0.7.0...v1.0.0) (2024-08-26) ### ⚠ BREAKING CHANGES * use chainhook to listen for chain events instead of a direct stacks api connection (#200) ### Features * convert data: image uris into image files and upload to cdn ([#245](#245)) ([903b0aa](903b0aa)) ### Bug Fixes * catch econnreset errors ([#247](#247)) ([51347d6](51347d6)) * set maximum job timeout ([#244](#244)) ([3444917](3444917)) * take only first page of gif images ([#241](#241)) ([334f8c5](334f8c5)) * use bignumber to handle FT supplies ([#239](#239)) ([053d622](053d622)) * use google cloud library for image uploads ([#238](#238)) ([c7f1b43](c7f1b43)) * use prometheus port configured in ENV ([c769d29](c769d29)) ### Code Refactoring * use chainhook to listen for chain events instead of a direct stacks api connection ([#200](#200)) ([2ddb2c7](2ddb2c7)), closes [#229](#229) [#232](#232) [#233](#233) [#234](#234) [#235](#235) [#236](#236)

blockstack-devops · 2024-08-26T22:04:59Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

rafaelcr added 22 commits June 21, 2023 16:36

chore: add chainhook client

05c8341

chore: single predicate draft

164afb6

chore: merge develop

f83a42f

chore: merge develop 2

d2588fa

fix: remove old code

074344a

fix: test builds

367b648

test: get contract abi

824485f

test: first sip-019 notif

0ce8947

test: print event chainhook observer

3bbd94e

feat: consume deployments

7a7fb5f

test: chain tip

f4b95c8

fix: unify chainhook events processors

9843195

feat: rollback contract deployments

9a1c06e

feat: rollback sft mint

b080190

chore: draft for sip-019 storage

a4f6c60

Merge branch 'develop' into refactor/chainhooks

21b7317

feat: keep sip-019 notifs

2957fd1

fix: notifications table inserts

f7ad5c0

chore: merge develop

5f9df71

fix: update code

6a78aa4

fix: some tests

5705fc9

fix: admin rpc

05b4921

rafaelcr temporarily deployed to Preview March 3, 2024 17:35 — with GitHub Actions Inactive

This was linked to issues Mar 3, 2024

NFT contracts with dynamic get-last-token-id values are not supported #140

Closed

use chainhooks instead of SmartContractMonitor and BlockchainImporter #146

Closed

support FTs with variable token supply #196

Closed

rafaelcr mentioned this pull request Mar 4, 2024

Collecting issues with token metadata for "tokensoft" stackswap launchpad tokens #199

Closed

rafaelcr linked an issue Mar 4, 2024 that may be closed by this pull request

Collecting issues with token metadata for "tokensoft" stackswap launchpad tokens #199

Closed

fix: rollback contracts and sip-013 mints

42e5a7c

test: zero token nft

40a4fdf

rafaelcr temporarily deployed to Preview July 19, 2024 16:27 — with GitHub Actions Inactive

smcclellan requested a review from tippenein July 22, 2024 15:59

rafaelcr added 3 commits August 15, 2024 11:04

fix: store chainhook predicate in local file for reuse after restart (#…

16e956e

…233) fix: add tests

Merge branch 'develop' into refactor/chainhooks

f02be17

rafaelcr temporarily deployed to Preview August 15, 2024 20:15 — with GitHub Actions Inactive

tippenein approved these changes Aug 16, 2024

View reviewed changes

rafaelcr temporarily deployed to Preview August 20, 2024 19:40 — with GitHub Actions Inactive

chore: upgrade to node 20 (#235)

4489c50

rafaelcr temporarily deployed to Preview August 20, 2024 19:54 — with GitHub Actions Inactive

feat: provide detailed error responses for tokens with user/contract …

f88f9c3

…errors (#236) * chore: demote httperror to usererror * fix: add errors * fix: custom messages

rafaelcr temporarily deployed to Preview August 20, 2024 22:40 — with GitHub Actions Inactive

rafaelcr requested a review from zone117x August 20, 2024 22:46

zone117x reviewed Aug 21, 2024

View reviewed changes

src/chainhook/server.ts Outdated Show resolved Hide resolved

zone117x reviewed Aug 21, 2024

View reviewed changes

zone117x approved these changes Aug 21, 2024

View reviewed changes

feat: add testnet support

df6317e

rafaelcr temporarily deployed to Preview August 21, 2024 15:50 — with GitHub Actions Inactive

rafaelcr merged commit 2ddb2c7 into develop Aug 21, 2024
8 of 9 checks passed

rafaelcr deleted the refactor/chainhooks branch August 21, 2024 15:54

blockstack-devops added the released on @beta label Aug 21, 2024

blockstack-devops added the released label Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor!: use chainhook to listen for chain events instead of a direct stacks api connection #200

refactor!: use chainhook to listen for chain events instead of a direct stacks api connection #200

rafaelcr commented Mar 3, 2024 •

edited

Loading

github-actions bot commented Mar 3, 2024 •

edited

Loading

zone117x Aug 21, 2024

rafaelcr Aug 21, 2024

zone117x Aug 21, 2024 •

edited

Loading

rafaelcr Aug 21, 2024

zone117x Aug 21, 2024

rafaelcr Aug 21, 2024

zone117x left a comment

blockstack-devops commented Aug 21, 2024

blockstack-devops commented Aug 26, 2024

refactor!: use chainhook to listen for chain events instead of a direct stacks api connection #200

refactor!: use chainhook to listen for chain events instead of a direct stacks api connection #200

Conversation

rafaelcr commented Mar 3, 2024 • edited Loading

github-actions bot commented Mar 3, 2024 • edited Loading

zone117x Aug 21, 2024

Choose a reason for hiding this comment

rafaelcr Aug 21, 2024

Choose a reason for hiding this comment

zone117x Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

rafaelcr Aug 21, 2024

Choose a reason for hiding this comment

zone117x Aug 21, 2024

Choose a reason for hiding this comment

rafaelcr Aug 21, 2024

Choose a reason for hiding this comment

zone117x left a comment

Choose a reason for hiding this comment

blockstack-devops commented Aug 21, 2024

blockstack-devops commented Aug 26, 2024

rafaelcr commented Mar 3, 2024 •

edited

Loading

github-actions bot commented Mar 3, 2024 •

edited

Loading

zone117x Aug 21, 2024 •

edited

Loading