Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running time #21

Open
rubenbucio opened this issue Apr 6, 2021 · 4 comments
Open

Running time #21

rubenbucio opened this issue Apr 6, 2021 · 4 comments

Comments

@rubenbucio
Copy link

Hi!

I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...

It has been running for over 27 hours now, is it normal? something is corrupted?

Used command:
python dvf.py -i test/CRC_meta.fa -l 1000 -c 14

I have:

  • h5py 2.10.0
  • keras 2.4.3
  • theano 1.0.5 (1.0.3 was not available for python 3.8)
@jessieren
Copy link
Owner

jessieren commented Apr 6, 2021 via email

@rubenbucio
Copy link
Author

rubenbucio commented Apr 6, 2021 via email

@jessieren
Copy link
Owner

jessieren commented Apr 6, 2021 via email

@jsgounot jsgounot mentioned this issue Jan 5, 2022
Open
@merytouceda
Copy link

Hi! I have the same issue as rubenbucio here. I am running deepvirfinder on my University's HPC, this is the code I am using to send the job:

!/bin/bash

#SBATCH --job-name=deepvirfinder
#SBATCH --output=deepvirfinder.out
#SBATCH --account=gornish
#SBATCH --mail-type=ALL
#SBATCH --mail-user=mtoucedasuarez@hpc.arizona.edu
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=5gb
#SBATCH --time=24:00:00


module load anaconda
source ~/.bashrc && conda activate
conda activate deepvirfinder


for sample in `awk '{print $1}' /xdisk/barberan/mtoucedasuarez/landuse/s_regex.txt`
do
   if [ "$sample" == "sample" ]; then continue; fi
   
   python /groups/barberan/software/DeepVirFinder/dvf.py -i /xdisk/barberan/mtoucedasuarez/landuse/assembly/contigs/${sample}_contig.fa -o /xdisk/barberan/mtoucedasuarez/landuse/virus/deepvirfinder/ -l 1500 -c 8

done

This is the error I get:

2022-12-02 15:07:22.159336: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-02 15:07:22.372586: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:22.372630: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-12-02 15:07:23.834156: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834282: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834297: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2022-12-02 15:07:25.996780: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-12-02 15:07:25.997115: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997129: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997215: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 520.61.5
2022-12-02 15:07:25.997265: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 520.61.5
2022-12-02 15:07:25.997277: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 520.61.5
2022-12-02 15:07:25.997590: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
1. Loading Models.
   model directory /groups/barberan/software/DeepVirFinder/models
2. Encoding and Predicting Sequences.
   processing line 1
   processing line 18044
Dec 03 15:07:19.999226 29969 slurmstepd   0x2b6d39ca6700: error: *** JOB 1468064 ON i16n0 CANCELLED AT 2022-12-03T15:07:19 DUE TO TIME LIMIT ***

Is there anything I can do to solve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jessieren @merytouceda @rubenbucio