[doc] added documentation to chunk and chunk manager

[FrankLeeeee]

While I read through the source for chunk.py, I found it lacks documentation so I just added it for better readability.

[FrankLeeeee]

Between, there are some APIs which I felt misleading in terms of semantics:

1. chunk.is_free should be indeed is_empty?
2. chunk.access looks more like chunk.synchronize to me.

@ver217

[ver217]

Between, there are some APIs which I felt misleading in terms of semantics:

1. chunk.is_free should be indeed is_empty?
2. chunk.access looks more like chunk.synchronize to me.

@ver217

1. Yes
2. access include moving data to CUDA ( and broadcast). Broadcast is not always needed, since ZeRO may not be applied.

[FrankLeeeee]

According to the code below, broadcast is always called when chunk.access() is invoked.

def access(self) -> None:
        """
        Broadcast the chunk to synchronize the tensors across data parallel processes.
        """
        # recover the chunk on non-owner processes
        # and broadcast the chunk from the source to all processes
        if not self.is_src_rank:
            self.data.storage().resize_(self.size)
        self.data.data = self.data.to(get_current_device())
        dist.broadcast(self.data, self.global_src_rank, group=gpc.get_group(ParallelMode.DATA))

        # update tensor meta info
        self._update_tensors_ptr()
        if not self.is_src_rank:
            self._update_tensors_state(TensorState.HOLD, prev_state=TensorState.FREE)

[ver217]

According to the code below, broadcast is always called when chunk.access() is invoked.

def access(self) -> None:
        """
        Broadcast the chunk to synchronize the tensors across data parallel processes.
        """
        # recover the chunk on non-owner processes
        # and broadcast the chunk from the source to all processes
        if not self.is_src_rank:
            self.data.storage().resize_(self.size)
        self.data.data = self.data.to(get_current_device())
        dist.broadcast(self.data, self.global_src_rank, group=gpc.get_group(ParallelMode.DATA))

        # update tensor meta info
        self._update_tensors_ptr()
        if not self.is_src_rank:
            self._update_tensors_state(TensorState.HOLD, prev_state=TensorState.FREE)

Oh, yes. But chunk_manager.access_chunk() don't always broadcast.

[FrankLeeeee]

@ver217 I have replaced is_free with is_empty. As for the access API, not a big deal but perhaps you can give a clearer name as access prompts the user to get some value in return and does not tell what is done by this API.

[ver217]

@ver217 I have replaced is_free with is_empty. As for the access API, not a big deal but perhaps you can give a clearer name as access prompts the user to get some value in return and does not entail what is done by this API.

OK