feat: adding google malware detection by jsdevel · Pull Request #36 · storacha/reads

jsdevel · 2022-08-28T08:11:42Z

Test ipfs malware sample from google with lookup and evaluate.

vasco-santos

Thanks for the first iteration @jsdevel

Let's please create a package from the start as previously talked and specified in https://hackmd.io/Y1vEmm9qTBmQU9Wzc3PY2w. Once we hook up tests, KV to store results / KV Locker it will be a bunch of logic that should be isolated. Also, we will want the logs for metrics + trigger badbits mail

For the package, we want a worker quite similar to this one with:

wrangler.toml
index.js
cors.js
error-handler.js
env.js

vasco-santos

Thanks for the new iteration Joe. Added some initial feedback even though I know this is still WIP 😉

vasco-santos

Thanks for the new iteration. Glad things are shaping up. A couple of general comments I could not have a place to add in review:

github actions workflow for cid verifier package (can be based on edge-gateway
CI is having a bad time, looks like pnpm lockfile should be commited? perhaps run on root dir pnpm clean && pnpm i

Also, this is first set of comments and I did not get into the actual code for CID verification. Just all the boilerplate around

vasco-santos

Extra thoughts, we should be good after this. Good job

jsdevel · 2022-09-02T00:05:37Z

Thanks for the new iteration. Glad things are shaping up. A couple of general comments I could not have a place to add in review:

github actions workflow for cid verifier package (can be based on edge-gateway

CI is having a bad time, looks like pnpm lockfile should be commited? perhaps run on root dir pnpm clean && pnpm i

Also, this is first set of comments and I did not get into the actual code for CID verification. Just all the boilerplate around

i saw that error. i checked in the pnpm lock file. i think locally i'm using a later version of pnpm and that's causing it.

jsdevel · 2022-09-08T22:38:35Z

New set of changes. Please also add README file, gather a copy from edge-gateway and basically need specific secrets for here specified there

i'll add a README once the approach is finalized

jsdevel · 2022-09-08T20:53:25Z

+export interface EnvInput {
+  ENV: string;
+  DEBUG: string;
+  GOOGLE_EVALUATE_SAFE_CONFIDENCE_LEVELS: Array<string>;


i had this in case there were other levels we'd want to configure other than SAFE

jsdevel · 2022-09-08T21:10:27Z

+  .get('/verification', withCorsHeaders(verificationGet))
+  .post('/verification', withCorsHeaders(verificationPost))


we can go over this in our call

vasco-santos · 2022-09-09T08:43:53Z

+export interface EnvInput {
+  ENV: string;
+  DEBUG: string;
+  GOOGLE_EVALUATE_SAFE_CONFIDENCE_LEVELS: Array<string>;


Ok, I got it thanks, this also needs to go to docs. And added in wrangler toml for other environments

* Moving DENYLIST logic to cid-verifier from edge-gateway

jsdevel

Hoping you can get this sample.html test to work.

jsdevel · 2022-09-10T02:49:59Z

  t.is(await response.text(), 'Hello dot.storage! 😎')
 })
+
+test('Sends HTML files to cid-verifier', async (t) => {


@vasco-santos i'm having trouble getting this to work. can you take a look?

vasco-santos

LGTM 🎉

jsdevel force-pushed the adding-google-malware-detection branch from 0fe2ded to 1b9c179 Compare August 28, 2022 08:13

jsdevel changed the title ~~Adding google malware detection.~~ feat: Adding google malware detection. Aug 28, 2022

jsdevel changed the title ~~feat: Adding google malware detection.~~ feat: adding google malware detection Aug 28, 2022

jsdevel force-pushed the adding-google-malware-detection branch from 1b9c179 to e6f4227 Compare August 28, 2022 08:14

vasco-santos suggested changes Aug 29, 2022

View reviewed changes

Comment thread packages/edge-gateway/src/gateway.js Outdated

Comment thread packages/edge-gateway/src/gateway.js Outdated

Comment thread packages/edge-gateway/src/gateway.js Outdated

vasco-santos reviewed Aug 30, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from 9f4a736 to c3dd17c Compare September 1, 2022 09:13

vasco-santos previously requested changes Sep 1, 2022

View reviewed changes

vasco-santos reviewed Sep 1, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 16 times, most recently from de4735a to 57a0953 Compare September 7, 2022 07:01

vasco-santos reviewed Sep 7, 2022

View reviewed changes

Comment thread packages/cid-verifier/package.json Outdated

Comment thread packages/cid-verifier/src/verification.js Outdated

jsdevel force-pushed the adding-google-malware-detection branch from 57a0953 to cc9b4f3 Compare September 7, 2022 22:20

storacha deleted a comment from vasco-santos Sep 7, 2022

jsdevel commented Sep 8, 2022

View reviewed changes

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from 8833a3f to 3960f33 Compare September 9, 2022 04:05

jsdevel commented Sep 9, 2022

View reviewed changes

Comment thread packages/cid-verifier/src/verification.js

vasco-santos suggested changes Sep 9, 2022

View reviewed changes

vasco-santos reviewed Sep 9, 2022

View reviewed changes

Comment thread packages/cid-verifier/src/verification.js Outdated

jsdevel force-pushed the adding-google-malware-detection branch 2 times, most recently from bd6d31c to 7d51262 Compare September 9, 2022 19:15

feat: adding google malware detection

65a107e

* Moving DENYLIST logic to cid-verifier from edge-gateway

jsdevel force-pushed the adding-google-malware-detection branch from 7d51262 to 65a107e Compare September 10, 2022 02:48

jsdevel commented Sep 10, 2022

View reviewed changes

test: requests to cid verifier with html response trigger verification

f3c6b84

vasco-santos force-pushed the adding-google-malware-detection branch from 6114aa7 to f3c6b84 Compare September 12, 2022 10:02

vasco-santos added 2 commits September 12, 2022 12:07

chore: add retries to kv

92978b8

chore: remove hearbeat

27c095f

vasco-santos approved these changes Sep 12, 2022

View reviewed changes

vasco-santos merged commit bacaeae into main Sep 12, 2022

vasco-santos deleted the adding-google-malware-detection branch September 12, 2022 10:25

This was referenced Sep 12, 2022

chore(main): release edge-gateway 1.2.0 #54

Merged

chore(main): release cid-verifier 1.0.0 #55

Merged

github-actions Bot mentioned this pull request Nov 14, 2024

chore(main): release cid-verifier 1.2.1 #195

Open

		.get('/verification', withCorsHeaders(verificationGet))
		.post('/verification', withCorsHeaders(verificationPost))

Conversation

jsdevel commented Aug 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasco-santos left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasco-santos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasco-santos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasco-santos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsdevel commented Sep 2, 2022

Uh oh!

Uh oh!

Uh oh!

jsdevel commented Sep 8, 2022

Uh oh!

Uh oh!

jsdevel Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

jsdevel Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasco-santos Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsdevel left a comment

Choose a reason for hiding this comment

jsdevel commented Aug 28, 2022 •

edited

Loading

vasco-santos left a comment •

edited

Loading

vasco-santos Sep 9, 2022 •

edited

Loading