Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wip2.0 #21

Closed
wants to merge 174 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
174 commits
Select commit Hold shift + click to select a range
4a08a04
Cleanup using standard
ninabreznik Aug 14, 2020
280b51c
Brotli logic added to EncoderDecoder
ninabreznik Aug 17, 2020
1c894f0
Add plan and publish for multiple feeds
ninabreznik Aug 23, 2020
1b38da9
Add custom derived topic key
ninabreznik Aug 26, 2020
1522dd3
refactoring part1 (WIP)
ninabreznik Sep 4, 2020
fd693fb
refactoring part2 closing streams
ninabreznik Sep 5, 2020
0634127
Add hoster (buggy)
ninabreznik Sep 7, 2020
8afc1f5
Add example for comm debugging
ninabreznik Sep 10, 2020
e3cbf57
All hoster logic added (buggy)
ninabreznik Sep 11, 2020
e44c674
Add logserver
ninabreznik Sep 14, 2020
4b057ba
Add logkeeper
ninabreznik Sep 14, 2020
1236125
Fixing bugs
ninabreznik Sep 20, 2020
e273cf6
Add selectAttestor fn to the chain
ninabreznik Sep 20, 2020
e4514c0
Update datdot-explorer
ninabreznik Sep 21, 2020
e282cdd
Add registrationForm for registering roles
ninabreznik Sep 22, 2020
319a21d
chain.js cleanup
ninabreznik Sep 22, 2020
a909be7
Update tryContract (add makeSets, size, hoster/attestor/encoder form)
ninabreznik Sep 29, 2020
f752a22
Cleanup chain.js
ninabreznik Oct 1, 2020
deec88f
Update registrationForm and add more test accounts
ninabreznik Oct 3, 2020
f0199f1
Refactor matchin logic in the chain
ninabreznik Oct 5, 2020
ac2a7e8
Update chain.js
ninabreznik Oct 7, 2020
bdb0ba7
Update chain.js
ninabreznik Oct 8, 2020
48e9d06
update chain - startHosting and attestorsJobs logic
ninabreznik Oct 8, 2020
80d11ed
Add isAvailableFromUntil check
ninabreznik Oct 9, 2020
cbc280b
Add scheduler and @TODO for stuff
ninabreznik Oct 14, 2020
7ef9180
Update tryContract logic
ninabreznik Oct 16, 2020
a7b622f
Remove splitPlanIntoOrders from TrContract
ninabreznik Oct 19, 2020
f4c7a64
Update getProviders
ninabreznik Oct 19, 2020
979cf7b
Refactor makeContracts, notifyProviders, getProviders
ninabreznik Oct 19, 2020
ad84406
Refactor makeDraftContracts and activateContracts + related
ninabreznik Oct 21, 2020
d7337bb
Update isAvailableFromUntil
ninabreznik Oct 21, 2020
0750566
Fix a bug in isAvailableFromUntil
ninabreznik Oct 22, 2020
e71b4ea
Add dropHosting and DB.hostings
ninabreznik Oct 23, 2020
b93d393
Bug fixes
ninabreznik Oct 24, 2020
8e1b0a6
Verifiy plan validity in publishPlan & update scheduleChallenges
ninabreznik Oct 25, 2020
4104e9b
Fix bug with challenges
ninabreznik Oct 31, 2020
8f835c2
Add provider jobs
ninabreznik Nov 1, 2020
571380d
Update add/remove jobs, hasCapacity etc.
ninabreznik Nov 2, 2020
ac19008
Add todo
ninabreznik Nov 2, 2020
b5e86ce
Add economics
ninabreznik Nov 2, 2020
d7d8e4f
Cleanup dates comparisons and emitDropHosting logic
ninabreznik Nov 3, 2020
07fedb0
Challenges and provider selection cleanup
ninabreznik Nov 4, 2020
58a43ba
Fix log issue in removeJob
ninabreznik Nov 4, 2020
9608a95
Refactoring + adding amendments - buggy
ninabreznik Nov 10, 2020
97df82a
Update tryNextContractJob
ninabreznik Nov 10, 2020
c729002
Update doesQualify
ninabreznik Nov 11, 2020
5219b76
Refactor user object and chainAPI
ninabreznik Nov 12, 2020
fe43e68
Update form.capacity check with user[role].capacty
ninabreznik Nov 12, 2020
ad4bc8e
Fix attestor reporting and replace hostingStarts with amendmentReport
ninabreznik Nov 19, 2020
3d20095
Replace dates with block numbers
ninabreznik Nov 21, 2020
c0412ce
Refactoring
ninabreznik Nov 23, 2020
bbc7dac
Fix getIdentity warning
RangerMauve Nov 23, 2020
265b96c
Merge branch 'wip' of github.com:playproject-io/datdot-service into wip
RangerMauve Nov 23, 2020
4483986
Fix storage reuse in hoster remove
RangerMauve Nov 23, 2020
c7c3e24
Fix p2plex and hoster storage + add stack trace
ninabreznik Nov 24, 2020
a0a9ac7
storageChallenge debugging
ninabreznik Nov 26, 2020
c111ce4
Refactoring with @serapath - round 1
ninabreznik Dec 23, 2020
da1e269
updating thee specs
ninabreznik Jan 3, 2021
20cb770
wip
serapath Feb 17, 2021
1849a71
Update up to publishPlan
ninabreznik Mar 28, 2021
b26c13f
Fixing contract/amendment logic
ninabreznik Apr 1, 2021
b177f73
Replace p2plex with hyperbeam and hypercores
ninabreznik Apr 7, 2021
2318aa4
Wire up attestor and hoster and store encoded data
ninabreznik Apr 8, 2021
a954742
Maybe fixed attestor to hosters and start hosting
ninabreznik Apr 9, 2021
e95171e
Maybe fixed attestor to hosters and start hosting
ninabreznik Apr 9, 2021
c1fce31
finish hosting setup refactoring for attestor
ninabreznik Apr 12, 2021
3b57783
Bug fix on hosting_setup + starting storage challenges
ninabreznik Apr 14, 2021
5ac51d5
Clean up storage proof basic logic
ninabreznik Apr 20, 2021
1dc80e4
Update notes
ninabreznik Apr 20, 2021
abb2fbf
Fix big with activate amendment and try next amendment
ninabreznik Apr 20, 2021
a16d0be
Refactoring Encoder account logic
ninabreznik Apr 22, 2021
3109e38
Move hoster logic to service.js and move helper modules to src/node_m…
ninabreznik Apr 23, 2021
4ea8a2b
Fix bug in loadUser & debugging signature verification error
ninabreznik Apr 25, 2021
61d159a
Update syncing logic for hoster clone feed
ninabreznik Apr 25, 2021
3f99470
Add ext message for storag challenge
ninabreznik Apr 26, 2021
6fd49d1
merge account and vault
ninabreznik Apr 27, 2021
eeea8c7
New worker role
ninabreznik Apr 28, 2021
b725b11
Move crypto logic to datdot-crypto
ninabreznik Apr 29, 2021
9b0cd29
Verfify signed event and rename roles and tasks
ninabreznik Apr 30, 2021
ee37eda
Move encoder and hoster logic to roles/[role]
ninabreznik May 4, 2021
5c1a409
remove service and serviceAPI and move hoster logic to roles/hoster
ninabreznik May 5, 2021
fa4f79c
fix encoder signature aka proof verification
ninabreznik May 8, 2021
9d8ea72
fix sparse replication
ninabreznik May 8, 2021
d6792f1
Update unique encoding to compress data + encoder_pos
ninabreznik May 10, 2021
b529c1e
Replace encoder_pos with encoder_id in unique signature
ninabreznik May 11, 2021
6bb5296
Merkle verify chunks (audit) before storing
ninabreznik May 11, 2021
cb8a37f
get, compare and send latest_root_signature when needed
ninabreznik May 13, 2021
46a55f8
Partial merkle verificiation
ninabreznik May 16, 2021
39b1fb9
verify-chunk
ninabreznik May 18, 2021
4141ad4
implement merkle verify for hoster and attester (no swarming)
ninabreznik May 19, 2021
b7435ee
First draft for merkle verification
ninabreznik May 23, 2021
06d0487
merkle verification working
ninabreznik May 24, 2021
150f400
Add merkle verification to the chain.js + create verify-chunk-data mo…
ninabreznik May 25, 2021
6ba134f
update datdot-codec and fix naming
ninabreznik May 28, 2021
cbdefe5
Refactor data object from encoder to attestor and to hoster + getting…
ninabreznik May 31, 2021
d725c21
Fix bug for verify_signature in submitStorageChallenge
ninabreznik May 31, 2021
933b0fd
switch to nonce amendment_id/encoder_pos + move ext message with host…
ninabreznik Jun 2, 2021
20ace6a
add hoster signing unique_el + verification by attester and the chain
ninabreznik Jun 3, 2021
2b3a901
Fix bug for getting ext message in storage challenge
ninabreznik Jun 4, 2021
9485b48
Partially refined perf challenge logic
ninabreznik Jun 9, 2021
b090fce
Optimize storage challenges
ninabreznik Jun 13, 2021
d7f88ab
Refactor logger, scheduler and chain
ninabreznik Jun 13, 2021
45971c8
add mempool and refactor blockgenerator
ninabreznik Jun 15, 2021
b25f7b0
REFACTOR - logging, directory structure and filenames
serapath Jun 19, 2021
09884e7
REFACTOR - logging, directory structure and filenames - part 2
serapath Jun 24, 2021
12f776b
add todo list file
serapath Jun 24, 2021
d184752
factor out research concept folder
serapath Jun 24, 2021
b9a0a02
refactor logging
serapath Jun 25, 2021
36a442b
fix logging bug
serapath Jun 25, 2021
9d0149d
fix log + set blockinterval to 5 seconds
serapath Jun 25, 2021
db7b71c
improve progress and error logging
serapath Jun 25, 2021
7194132
improve progress and error logging - part 2
serapath Jun 25, 2021
194d5ab
Refactor schedule storage challenge
ninabreznik Jun 25, 2021
53ef2da
Update create_signing_keypair
ninabreznik Jun 25, 2021
dc7c0df
Update package.json and datdot-crypto
ninabreznik Jun 26, 2021
a7778db
Fix schedule storage challenge bug
ninabreznik Jun 28, 2021
28e2010
Add status to challenges.storage and bug fixes
ninabreznik Jun 28, 2021
b6000d4
replace hoster.challenges.storage with hoster.storage_challenges
ninabreznik Jun 28, 2021
5643b36
first iteration datdot-swarm
ninabreznik Jul 19, 2021
c63bfee
datdot-swarm partial
ninabreznik Jul 23, 2021
a68d909
add @hyperswarm/dht#v5 to dependencies
serapath Jul 23, 2021
844fa1c
fix datdot-explorer
serapath Jul 23, 2021
8432f24
remove duplicated hypercore-append module
ninabreznik Jul 24, 2021
df1b5a6
refactor datdot-explorer
serapath Jul 24, 2021
102321a
refactor codes for logs
Jul 24, 2021
fad4a3a
update view and styles
Jul 26, 2021
5a2ac8e
update view and styles
Jul 26, 2021
f73f17c
remove datdot-ui-logs, add datdot-terminal, update layout and style
Jul 27, 2021
e239f72
remove status and button
Jul 28, 2021
fa0b98d
update terminal
Jul 28, 2021
dd2307b
update
Jul 30, 2021
ed1029b
update terminal styles
Jul 30, 2021
8955e21
update
Jul 30, 2021
0f4ca48
Fix sig check in amendment_report
ninabreznik Aug 2, 2021
10cd96e
replace amendment_follow up with amendment_timeout
ninabreznik Aug 2, 2021
3997924
Rename jobs to tasks
ninabreznik Aug 3, 2021
a54f124
Merge pendingAmendments and attestorJobs and rename jobs to tasks
ninabreznik Aug 4, 2021
bf29130
add rating and balance to remove_task_and_reward
ninabreznik Aug 5, 2021
dc48de7
store update
Aug 5, 2021
9a22435
update terminal
Aug 6, 2021
035fc7b
Add unbook_and_reward
ninabreznik Aug 7, 2021
8a43ee3
Merge branch 'wip' of https://github.com/playproject-io/datdot-servic…
ninabreznik Aug 7, 2021
fa39cd8
Add grade logic for reward hoster
ninabreznik Aug 10, 2021
5467d2a
Bug fixing for StorageChallenge
ninabreznik Sep 13, 2021
7ed6dea
Update hosting-plan networking behavior
ninabreznik Sep 14, 2021
20340b4
Add finished(stream)
ninabreznik Sep 14, 2021
5a3a618
add dropdown and expanded button
Sep 19, 2021
d896b0a
Make load feed module
ninabreznik Sep 20, 2021
56e74fc
Merge branch 'wip' of https://github.com/playproject-io/datdot-servic…
ninabreznik Sep 20, 2021
ebb1a9a
Fix hoster ext bug
ninabreznik Sep 28, 2021
50d4f7b
Bug fixing
ninabreznik Sep 30, 2021
cd4ad87
Update brotli path
ninabreznik Sep 30, 2021
67a5a93
removed node_mouldes
Sep 30, 2021
20a5fc4
add .gitignore
Sep 30, 2021
96e4a53
remove gitignore files
ninabreznik Sep 30, 2021
996f671
Update package.json
ninabreznik Sep 30, 2021
45f2e0c
Add update feed storage
ninabreznik Oct 2, 2021
0ea4bcf
Make update-hoster-storage module
ninabreznik Oct 3, 2021
4987475
folk datdot-service and modify datdot-explorer in 40 mins
Oct 3, 2021
505f111
Add bootstrapping nodes
ninabreznik Oct 5, 2021
0889487
Transition to hypercore 10 - wip
ninabreznik Oct 23, 2021
55e7ac0
Back to hypercore 9.10.0
ninabreznik Oct 23, 2021
2424bcc
Fix getting data from hoster in storage challenge
ninabreznik Oct 23, 2021
f26bc5d
Update intercepter - get- and putData, add account.cache logic, updat…
ninabreznik Oct 26, 2021
2b49cd1
Update load feed
ninabreznik Nov 1, 2021
7aa4ae7
load feed v1, buggy
ninabreznik Nov 2, 2021
8441ffe
refactor feed-store
ninabreznik Nov 11, 2021
d406dfd
Update feed-store
ninabreznik Nov 16, 2021
791b8d5
Implement new swarm per new feed on same topic
ninabreznik Nov 18, 2021
75379df
updated todos, fix boostrapper and user registration data
ninabreznik Oct 14, 2022
b8f85d2
Update polkadot related libraries
ninabreznik Oct 15, 2022
d495a5a
got discovery key callback
ninabreznik Oct 26, 2022
9a08e76
Verify proof for new plan and store signature
ninabreznik Nov 3, 2022
e31bd09
refactoring feed-store
ninabreznik Nov 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
/@todo/*
/node_modules/*
/package-lock.json
/pnpm-lock.yaml
/npm-debug.log

/yarn.lock
/yarn-error.log
.DS_Store
.DS_Store
Empty file modified LICENCE
100644 → 100755
Empty file.
Empty file modified README.md
100644 → 100755
Empty file.
10 changes: 0 additions & 10 deletions cli.js

This file was deleted.

37 changes: 37 additions & 0 deletions demo/bootstrapper.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
const DHT = require('@hyperswarm/dht')

module.exports = bootstrappers

async function bootstrappers ({ amount = 1, log = console.log.bind(console) } = {}) {
if ((!Number.isInteger(amount)) || amount < 2) amount = 1

const node0 = await make_node({ port: 10000, ephemeral: false, bootstrap: [] })
const info0 = info(node0)

const bootstrap_nodes = [{ host: info0.address.host, port: info0.address.port }]


for (var i = 0; i < amount; i++) {
const node = await make_node({ port: 10001 + i, ephemeral: false, bootstrap: bootstrap_nodes })
const meta = info(node)
bootstrap_nodes.push({ host: meta.address.host, port: meta.address.port })
}

log('bootstrap_nodes', ...bootstrap_nodes)
return bootstrap_nodes
}

async function make_node ({ port = 10000, ephemeral = true, bootstrap = [] }) {
const node = new DHT({ bootstrap, ephemeral, firewalled: false })
await node.bind(port)
return node
}

function info (node) {
const pk = node.defaultKeyPair.publicKey.toString('hex')
const sk = node.defaultKeyPair.secretKey.toString('hex')
// Your DHT routing id is hash(publicIp + publicPort) and will be autoconfigured internally.
const id = node.table.id.toString('hex')
const { host, port } = node.address()
return { id, pk, sk, address: { host, port } }
}
21 changes: 21 additions & 0 deletions demo/scenarios/1.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@

module.exports = [
{ "name": "alice", "behavior": ["register_chain_account", "subscribe_to_plan", "offer_service"] },
{ "name": "eve", "behavior": ["register_chain_account", "author_hypercore"] },
{ "name": "bob", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "charlie", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "dave", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "janice", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "ferdie", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "one", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "two", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "tom", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "ashley", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "emma", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "joe", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "anna", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "paul", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "rachel", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "alfie", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "poppy", "behavior": ["register_chain_account", "offer_service"] },
]
11 changes: 11 additions & 0 deletions demo/scenarios/2.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

module.exports = [
{ "name": "iva", "behavior": ["author_hypercore"] },
{ "name": "luigi", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "rok", "behavior": ["register_chain_account", "subscribe_to_plan"] },
{ "name": "bianca", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "bent", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "felix", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "elfi", "behavior": ["register_chain_account", "offer_service"] },
{ "name": "clemens", "behavior": ["register_chain_account", "offer_service"] },
]
3 changes: 3 additions & 0 deletions demo/scenarios/3.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
// TODO:
// bob tries to register as a offer_hosting twice
// author has no additional actions
99 changes: 99 additions & 0 deletions demo/simulation.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
process.env.DEBUG = '*,-hypercore-protocol'
const spawn = require('cross-spawn')
const path = require('path')
const bootstrapper = require('./bootstrapper.js')

run()

async function run (){

const bootstrap_nodes = await bootstrapper({ amount: 10 })

console.log({bootstrap_nodes})

await new Promise(resolve => setTimeout(resolve, 250))

const [scenarioName, flag] = process.argv.slice(2)
process.env.REPORT = `*
+**,-
+*:*:**@bar#xxx;

alice:chain

asdf@fooo#x
asdf@fooo
asdf#x
@fooo#x
asdf
@fooo
#x
`

if (!scenarioName) return console['log'](`
USAGE: npm run simulation <scenario-filename>

(To customise logs,
edit "demo/simulation.js"
and change "process.env.REPORT")

SCENARIOS:
./demo/scenarios/1.js
./demo/scenarios/2.js
./demo/scenarios/foo.js

EXAMPLE 1
> npm run simulation 1

EXAMPLE 2
> npm run simulation foo.js
`)

const scenario = require(`./scenarios/${scenarioName}`)

const all = [process]
const url = 'ws://localhost'

const config = JSON.stringify({ chain: [url, 3399], chat: [url, 6697], bootstrap: bootstrap_nodes })

if (flag === '--production') {
throw new Error('"--production" is currently unsuported')
// const command1 = path.join(__dirname, '../datdot-substrate/target/release/datdot-node')
// const child = spawn(command1, ['--dev'], { stdio: 'inherit' })
// all.push(child)
} else {
const args1 = [path.join(__dirname, '../src/node_modules/datdot-node-javascript/chain.js'), config, /*logkeeper*/9001]
const chain = spawn('node', args1, { stdio: 'inherit' })
all.push(chain)
}

const args2 = [path.join(__dirname, '../src/node_modules/_chat/server.js'), config, /*logkeeper*/ 9002]
const chatserver = spawn('node', args2, { stdio: 'inherit' })
all.push(chatserver)

const users = scenario
const PORTS = [9001, 9002]
const file = path.join(__dirname, '../src/node_modules/datdot-simulate-user/user.js')
for (var i = 0, len = users.length; i < len; i++) {
const scenario = JSON.stringify(users[i])
const logkeeper_port = 9003 + i
PORTS.push(logkeeper_port)
const args = [file, scenario, config, logkeeper_port]
const child = spawn('node', args, { stdio: 'inherit' })
all.push(child)
}

const args3 = [path.join(__dirname, '../src/node_modules/datdot-explorer/demo/registry.js'), JSON.stringify(PORTS)]
const logkeeper = spawn('node', args3, { stdio: 'inherit' })
all.push(logkeeper)

process.on('SIGINT', () => {
console['log']("\n Terminating all processes")
const [main, ...processes] = all
for (var i = 0, len = processes.length; i < len; i++) {
const child = processes[i]
child.kill()
}
console['log']('done')
main.exit(0)
})
}
39 changes: 0 additions & 39 deletions lab/scenarios/mvp-1.js

This file was deleted.

Loading