Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update plex job publisher spec for bacalhau v1.1 #692

Conversation

acashmoney
Copy link
Contributor

@acashmoney acashmoney commented Oct 5, 2023

For bacalhau v1.1 migration, we need to ensure PublisherSpec.Type is ipfs instead of estuary, which is deprecated as of v1.1

Steps to Test:

  1. export BACALHAU_API_HOST=bacalhau.prod.labdao.xyz
  2. Confirm server is v1.1.0 with bacalhau version
  3. go run main.go init -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
  4. Confirm PublisherSpec.Type is ipfs with bacalhau describe <job-id>

Should see a successful run like below.

go run main.go init -t QmZWYpZXsrbtzvBCHngh4YEgME5djnV5EedyTpc8DrK7k2 -i '{"protein": ["QmUWCBTqbRaKkPXQ3M14NkUuM4TEwfhVfrqLNoBB7syyyd/7n9g.pdb"], "small_molecule": ["QmViB4EnKX6PXd77WYSgMDMq9ZMX14peu3ZNoVV1LHUZwS/ZINC000019632618.sdf"]}' --scatteringMethod=dotProduct --autoRun=true -a test
Plex version (v0.10.4) up to date.
Pinned IO JSON CID: QmWhpHj9qWwHETVwCJtK1o5wXvxF8h7oVBxQQKpp4HkDXK
Created working directory: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/48c0acb6-7a59-4210-a49f-7f91c28e6150
Initialized IO file at: /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/48c0acb6-7a59-4210-a49f-7f91c28e6150/io.json
Extracting user ID from IO JSON...
Processing IO Entries
Starting to process IO entry 0 
Job running...
Bacalhau job id: 891dd10b-d920-4ca7-b741-53bbd3d86afb 
////🌱____////
Computed default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "8.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Theses can be inspected with 'ipfs swarm resources'.

Success processing IO entry 0 
Finished processing, results written to /Users/aakaash/Desktop/code/OPENLAB/plex/jobs/48c0acb6-7a59-4210-a49f-7f91c28e6150/io.json
Completed IO JSON CID: QmQRiwxsMxzcReALZEXC45rELaLdGwaqMviwKTYa4Xnq72

Running bacalhau describe on the job ID...

bacalhau describe 891dd10b-d920-4ca7-b741-53bbd3d86afb
Job:
  APIVersion: V1beta2
  Metadata:
    ClientID: c1cda6a8463a6bd832fa91a04cb2ab3704f7cc5a6761b0ba686781fd5768d114
    CreatedAt: "2023-10-05T16:38:29.150194654Z"
    ID: 891dd10b-d920-4ca7-b741-53bbd3d86afb
    Requester:
      RequesterNodeID: QmZkGQUCERDq8hWv67JkVurLmjXzRFYiMPoueBXb6FJR2P
      RequesterPublicKey: CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDqPOHv26xJPsCrPdslpwv07P5R5pAZLnFYfxcn/02ZoKg/3TfsqOrINGGW5XqFNgp5VZULTrkfQK0bsyPK2jrhw4dA21u4TbKwugRM3jA6kQvz+hj5HL9MT2NUFt2qvoCIXSXyDerLGLsNkD0HJwV75jsrICgP/58vsjJk/3EEggJRQnjznAq7m5dZb9zTCx1Em90Vh19tdSvSLKJFjtlJW/S+VCaj+4biMCsO30c51rsIH3kRDMrbniqLOy7IsGHAUhUgpmPYYELry/Om97hXPJ6FoDAnMUOVpcb1SaABZYp/1gt5h1wwTiE6XUvu+rjS48KSp+2YCCK/1zDIA4QVAgMBAAE=
  Spec:
    Annotations:
    - test
    - userId=0x1df13A64355BE17118a05C00dA9815DAa4ab1ea0
    Deal:
      Concurrency: 1
    Docker: {}
    EngineSpec:
      Params:
        Entrypoint:
        - /bin/bash
        - -c
        - /bin/bash -c "mkdir -p /tmp-inputs/tmp; mkdir -p /tmp-outputs/tmp; cp /inputs/*
          /tmp-inputs/tmp/; ls /tmp-inputs/tmp; cd /src && python /src/inference.py
          --config=/src/configs_clean/bacalhau.yml; mv /tmp-outputs/tmp/* /outputs/;
          mv /outputs/lig_equibind_corrected.sdf /outputs/7n9g_ZINC000019632618_docked.sdf;
          mv /tmp-inputs/tmp/*.pdb /outputs/;"
        EnvironmentVariables: null
        Image: ghcr.io/labdao/equibind:main@sha256:21a381d9ab1ff047565685044569c8536a55e489c9531326498b28d6b3cc244f
        Parameters: null
        WorkingDirectory: ""
      Type: docker
    Inputs:
    - CID: QmXURBWYWbphx9abyFRWotGUwXqesv5xdaAcro8aXLxHQu
      Path: /inputs
      StorageSource: ipfs
    Network:
      Type: None
    Outputs:
    - Name: outputs
      Path: /outputs
    PublisherSpec:
      Type: ipfs
    Resources:
      GPU: ""
    Timeout: 3600
    Wasm:
      EntryModule: {}
State:
  CreateTime: "2023-10-05T16:38:29.150194654Z"
  Executions:
  - ComputeReference: e-4f9dcb21-6862-4d33-9995-ae2d53b22335
    CreateTime: "2023-10-05T16:38:29.183447322Z"
    DesiredState: 2
    JobID: 891dd10b-d920-4ca7-b741-53bbd3d86afb
    NodeId: QmNmSwDk8zE6DYKyhnJrzLtgzscnXb8EreDHFsTTKV56UZ
    PublishedResults:
      CID: QmYPfPNZKUKK42hUATFDfz4Kz8GbQEgopiLNqdiHhuZ9o6
      StorageSource: ipfs
    RunOutput:
      exitCode: 0
      runnerError: ""
      stderr: |
        DGL backend not selected or invalid.  Assuming PyTorch for now.
      stderrtruncated: false
      stdout: |
        7n9g.pdb
        ZINC000019632618.sdf
        Setting the default backend to "pytorch". You can change it in the ~/.dgl/config.json file or export the DGLBACKEND environment variable.  Valid options are: pytorch, mxnet, tensorflow (all lowercase)
        [2023-10-05 16:38:43.853340] [ Using Seed :  1  ]

        Processing tmp: complex 1 of 1
        Trying to load /tmp-inputs/tmp/ZINC000019632618.sdf
        Docking the receptor /tmp-inputs/tmp/7n9g.pdb
        To the ligand /tmp-inputs/tmp/ZINC000019632618.sdf
        Writing prediction to /tmp-outputs/tmp/lig_equibind_corrected.sdf
        Saving predictions to runs/flexible_self_docking/predictions_RDKitFalse.pt
      stdouttruncated: false
    State: Completed
    Status: . execution completed
    UpdateTime: "2023-10-05T16:39:10.904681546Z"
    Version: 6
  JobID: 891dd10b-d920-4ca7-b741-53bbd3d86afb
  State: Completed
  TimeoutAt: "2023-10-05T17:38:29.150194654Z"
  UpdateTime: "2023-10-05T16:39:10.94856812Z"
  Version: 2

@linear
Copy link

linear bot commented Oct 5, 2023

@vercel
Copy link

vercel bot commented Oct 5, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated (UTC)
docs ⬜️ Ignored (Inspect) Oct 5, 2023 4:40pm

@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:40 — with GitHub Actions Inactive
@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:40 — with GitHub Actions Inactive
@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:40 — with GitHub Actions Inactive
@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:40 — with GitHub Actions Inactive
@acashmoney acashmoney changed the title update plex job publisher spec for v1.1 update plex job publisher spec for bacalhau v1.1 Oct 5, 2023
Copy link
Collaborator

@alabdao alabdao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:43 — with GitHub Actions Inactive
@acashmoney acashmoney temporarily deployed to ci October 5, 2023 16:43 — with GitHub Actions Inactive
@acashmoney acashmoney merged commit 26aa88c into main Oct 5, 2023
8 checks passed
@acashmoney acashmoney deleted the 691-lab-664-lab-662-update-bacalhau-job-publisher-to-ipfs-instead-of-estuary branch October 5, 2023 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[LAB-664] [LAB-662] update bacalhau job publisher to IPFS instead of Estuary
2 participants