[WIP] feat: market 2.0

[LexLuthr]

• Establish consensus on design
• Aggregate Indexing
• Aggregate IPNI
• Test contract
• PieceCID V2
  • Retrievals
  • Indexing
  • IPNI
  • UI - MK12 and MK20
  • MIgration to v2
  • Indexing Repair task
• Write deal validation logic tests
• MK2.0 UI
• Notification pathway
• Create mk20 test client
• Write documentation for MK2.0 standard
• Move PDP endpoints under market
• Move any other market related endpoint
• HTTP PUT UI page
• Testing
  • Devnet
  • Calibnet
  • Mainnet

Tests:

1. Make mk20 deal with all data source types
2. Force a mk12 and mk20 deal in a sector
3. Deal status
4. Test deal validation logic on each param
5. Test any other endpoints
6. Test UI
7. Deal cancellation
8. Test indexing and announcements
9. Bombard system with mk12 and mk20 deals and see if anything breaks
10. Payload Retrievals
11. Piece Retrievals
12. Payload Retrieval from aggregate piece
13. Sub piece from aggregate Retrievals
14. Fix index
15. Fix IPNI
16. V1 -> V2 migration of indexes
17. Bulk Restart mk20
18. Bulk remove mk20
19. PDP

[magik6k]

Just reviewing types for now, didn't look at much beyond that, but generally things make sense.

[LexLuthr]

Refactor

• Refactor Deal{} struct
• Figure out where to add market deal ID for PoRep
• Add PDP product
• Allow uploading data before data details
• Refactor chunk aggregation to work for PDP
• Set a limit to delete data if not finalized in certain time
• Create PDP pipeline
• Create method to update deal
• Update DB deal table to have complete, error columns for each product
• Create a clean up task for indexing/ipni based on expired sectors and completed PDP deals (deleteRoot)
• Refactor UI including PDP
• Remove unused PDP endpoints
• Move mk20 endpoints to directly under market
• Rename tables as required to differentiate between PoRep and PDP if required
• Figure out indexing when PoRep is later added to PDP deals (market piece deal table)
• Update retrievals to use market piece deal table instead of PDP workarounds.
• Figure out IPNI piece only announcement
• Figure out IPNI announcements when PoRep is added to PDP deal
• Should PDP support other data sources? Especially aggregate (PODSI) data?