Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to run initialise training #18

Closed
jordanparker6 opened this issue Jul 9, 2019 · 0 comments
Closed

Unable to run initialise training #18

jordanparker6 opened this issue Jul 9, 2019 · 0 comments

Comments

@jordanparker6
Copy link

I am having the following issue...

The model compiles and prints the following output. However, on model.fit() nothing happens, despite verbose mode being turned on.

When i look at my hardware utilisation, my GPU has memory allocated to the process however utilisation is 0-2%. On my CPU, only one core is getting worked by the process at 100% utilisation.

To test my tensorflow-gpu install, I ran the CNN example on tensorflow and got 20% GPU utilisation.

I don't think it is a preprocessing bottleneck as I load my training data into memory.

Thanks.

Code:
` bert_path = "https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1"
max_seq_length = 256

corpus = MyDocs("datasets/bbc/raw", bert_path, max_seq_length)

ids = []
masks = []
segment_ids = []
for id, mask, segment, label in corpus:
    ids.append(id)
    masks.append(masks)
    segment_ids.append(segment)
X = [ids, masks, segment_ids]

labels = corpus.labels
label_encoder = OneHotEncoder()
y = label_encoder.fit_transform(np.array(labels).reshape(-1, 1)).todense()

print('Building model...')
model = build_model(bert_path, max_seq_length)

print('Training model...')
history = model.fit(X, y,
                    validation_split=0.2,
                    epochs=1,
                    batch_size=1,
                    verbose=2,
                    use_multiprocessing=True)`

Output:

Building model...
W0709 21:57:53.871020 140194145126208 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0709 21:57:53.922768 140194145126208 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

Training model...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant