# what's the meaning of 'subdivisions' and how to set it's value? #224

Closed
opened this issue Oct 2, 2017 · 13 comments
Closed

# what's the meaning of 'subdivisions' and how to set it's value?#224

opened this issue Oct 2, 2017 · 13 comments

### yanchao12122 commented Oct 2, 2017

 how to set the value of subdivisions? [net] batch=64 subdivisions=8

### TheMikeyR commented Oct 11, 2017 • edited

 It's how many mini batches you split your batch in. Batch=64 -> loading 64 images for this "iteration". Subdivision=8 -> Split batch into 8 "mini-batches" so 64/8 = 8 images per "minibatch" and this get sent to the gpu for process. That will be repeated 8 times until the batch is completed and a new itereation will start with 64 new images. When batching you are averaging over more images the intend is not only to speed up the training process but also to generalise the training more. If your GPU have enough memory you can reduce the subdivision to load more images into the gpu to process at the same time. Hope this answer your question.

### yanchao12122 commented Oct 11, 2017

 @TheMikeyR what you explanation is so great, thank you very much

### ZHANGKEON commented Jul 19, 2018 • edited

 @pjreddie @TheMikeyR I am still confused of how the subdivision affect the update of moving mean and moving variance in batch norm. My understanding is that the mean and variance will be computed based on the mini-batches' size (e.g. 8) instead of the total batch size (e.g. 64). Hence, the mini-batches is the actual batch size used in batch normalization. Is my understanding right?

### keides2 commented Jul 27, 2018

 When batch = 64, subdivisions = 64, the mAP was 0.008%, but when we changed to batch = 64, subdivisions = 32, mAP improved to 97%. When the value of loss is around 0.1 in either case. I think that the false judgment ratio (FalsePositive) decreases, could you tell me why mAP improves?

### ZHANGKEON commented Jul 30, 2018 • edited

 By using a smaller subdivision, the mini-batch size for computing gradient increases. Hence, the computed gradient based on a larger mini-batch size gives a better optimization. I guess using a smaller mini-batch size will result in a local optimum and thus decrease accuracy.

### keides2 commented Jul 30, 2018

 @ZHANGKEON Can mini-batch size be the same value as epoch number? For example, for simplicity, if there are 320 images, batch = 64, subdivisions is 64, the epoch number is 5 (= 320/64), and to learn all images only 5 iterations. If subdivisions is 8, epoch number is 40 (= 320/8), and it can be learned 40 times. Here, mini-batch size can be considered to be 5 or 40.

### keides2 commented Jul 30, 2018

 I made a mistake. The mini batch size is batch / subdivision. batch = 64, subdivisions = 32 mini-batch size = 64/32 = 2 epoch = 320/2 = 160 batch = 64, subdivisions = 8 mini-batch size = 64/8 = 8 epoch = 320/8 = 40

### gustavovaliati commented Aug 17, 2018 • edited

 @keides2 , I hope I am not late. :) As far as I know, the common concept of epoch is different from what you have stated. Running your algorithm for 1 epoch means it has processed all the 320 samples a single time. Two epochs mean you have processed the whole dataset two times. In your example, for each epoch (processing 320 samples) your gonna need 5 iterations in batches of 64. dataset = 320 samples = 1 epoch batch = 64 subdivisions = mini-batches = 32 1 epoch = 5 iterations = 320/64 1 iteration = 32 mini-batches = 1 batch mini-batch size = 64/32 = 2 samples

### naren142 commented Aug 21, 2018

 How to train the yolo with 3 frames at a time? The input should be like first 3 frames(1,2,3) and then second 3 frames(2,3,4) like this?

### yuenbe commented Oct 4, 2018

 Hi, I've read the above comments but I don't really understand the purpose of defining a batch value and a subdivision value. I suppose the max_batches parameter refers to the maximum number of trained batches, so if I write something like ``````batch=64 subdivisions=2 max_batches=10000 `````` wouldn't it be the same as writing? ``````batch=32 subdivisions=1 max_batches=20000 `````` The mini-batch size would be same in both cases. Is there any difference in terms of computation speed and/or accuracy between the above configurations?

### humandotlearning commented Nov 28, 2018

 Hi, I've read the above comments but I don't really understand the purpose of defining a batch value and a subdivision value. I suppose the max_batches parameter refers to the maximum number of trained batches, so if I write something like ``````batch=64 subdivisions=2 max_batches=10000 `````` wouldn't it be the same as writing? ``````batch=32 subdivisions=1 max_batches=20000 `````` The mini-batch size would be same in both cases. Is there any difference in terms of computation speed and/or accuracy between the above configurations? actually the batch is where the gradients are updated. for instance i have batch =64 subdivision=8 then after the model has seen 64 images it would update its gradients and the subdivision is given to reduce the ram usage of gpu at one time. so for the above case the mini batch would be 68/8 = 8. that is 8 images are loaded in the gpu at one time.

### keides2 commented Dec 6, 2018

 @gustavovaliati , I was too late to notice. :) I understood that one epoch is to process all 320 samples once. 1 iteration = 32 mini batches = 1 batch Correctly, Mini batch size = batch / mini batches = 64/32 So, 1 iteration = mini batches * mini batch size = 32 * 64/32 = 1 batch is not it? Thankyou,

### jveitchmichaelis commented Dec 28, 2018 • edited

I think there some subtleties in the codebase which are causing confusion. Common terminology is:

Epoch = a pass through the full dataset
Batch training = update the weights after one epoch
Minibatch training = update the weights after a set number of samples have been seen

In darknet, the batch parameter is the minibatch size. The subdivision parameter controls how many samples will fit into RAM (minibatch/subdivision = samples loaded simultaneously per pass) - a mini-minibatch, if you like. A subdivision is not a minibatch in the conventional sense, because in darknet the weights are not updated every batch (here's where the naming is confusing) unless `subdivision == 1`.

If you look in the code, what gets printed during training is:

```i = get_current_batch(net);
int imgs = net->batch * net->subdivisions * ngpus;
...
printf("%ld: %f, %f avg, %f rate, %lf seconds, %d images\n", get_current_batch(net), loss, avg_loss, get_current_rate(net), what_time_is_it_now()-time, i*imgs);```

where the current batch is defined as:

```size_t get_current_batch(network *net)
{
size_t batch_num = (*net->seen)/(net->batch*net->subdivisions);
return batch_num;
}```

If we see:

`26132: 5.084459, 4.797126 avg, 0.001000 rate, 2.847984 seconds, 1672448 images`

That means the current batch is 26132 and my batch size is 64, which gives 1672448. However this is strange - this would suggest that either `net->subdivisions = 1` which it isn't, or that the number of `seen` images is wrong (it isn't - see further down), so.. the batch size is wrong?

On the face of it, something seems messed up - surely the current batch is `n_images/batch`. Why are we multiplying by `subdivisions`?

Some clarifying information from pjreddie here: https://groups.google.com/forum/#!topic/darknet/qZp5P0I4iGA

Yes exactly, it's a confusing implementation detail but subdivisions is just used to divide up the work and limit memory usage. If you have a batch size of 10 and subdivision of 2 your GPU will only process 5 images at a time but will update weights after 2 of the "new" batch size. Thus batch_size is sort of overloaded in the code, eventually I'll probably edit it to make it more clear.

It turns out that `net->batch` gets modified when the config file is loaded - here:

Line 664 in 61c9d02

 net->batch /= subdivs;

Since the subdivision is a clean divisor of the batch size, this means that `net->batch` is always an integer >= 1. When you later refer to `net->batch` in the code, it's really a smaller `net->batch/subdivision` and so you need to compensate later on. Pretty confusing!

Let's look at the training code:

```float train_network(network *net, data d)
{
assert(d.X.rows % net->batch == 0);
int batch = net->batch;
int n = d.X.rows / batch;

int i;
float sum = 0;
for(i = 0; i < n; ++i){
get_next_batch(d, batch, i*batch, net->input, net->truth);
float err = train_network_datum(net);
sum += err;
}
return (float)sum/(n*batch);
}

float train_network_datum(network *net)
{
*net->seen += net->batch;
net->train = 1;
forward_network(net);
backward_network(net);
float error = *net->cost;
if(((*net->seen)/net->batch)%net->subdivisions == 0) update_network(net);
return error;
}```

So when we train, we grab `batch` images, which is the scaled batch size. We increment `seen` by this amount.

Going into the datum function, we do a forward pass and a backward pass (so far so good), calculate the loss and then update the weights if we've seen enough images. So we need to compute the number of images divided by the smaller batch size, and then check if that's a multiple of the number of subdivisions. In a roundabout way, that makes sense.

In summary:

• Darknet overloads some terminology in the code, but the config parameters mostly make sense
• Batch size is the number of images you process before updating the network weights (i.e. a minibatch in other work)
• The batch size is divided by subdivisions when the network is loaded, this is how many images can fit into your GPU at once. In the code, net->batch really refers to that, not the minibatch size.
• An iteration is one mini batch (e.g. whatever you set batch to in the config file)
• Bear this in mind when reading through the source!