Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"fix client send empty gradients bug" #2654

Merged
merged 2 commits into from
Jun 29, 2017

Conversation

dzhwinter
Copy link
Contributor

In some miss use situation, SendGrads will get empty slice here, then errCh will block forever.

// SendGrads sends gradients to parameter servers for updating
// parameters.
func (c *Client) SendGrads(grads []Gradient) error {
  if len(grads) == 0 {
    log.Info("Send Empty Gradient")
    return nil
  }
  errCh := make(chan error, len(grads))
  for _, g := range grads {
    go func(g Gradient) {
      err := c.pservers[c.partition(g.Name)].Call("Service.SendGrad", g, nil)
      errCh <- err
    }(g)
  }

  recv := 0
  for err := range errCh {
    if err != nil {
      return err
    }

@@ -123,6 +123,10 @@ func (c *Client) FinishInitParams() error {
// SendGrads sends gradients to parameter servers for updating
// parameters.
func (c *Client) SendGrads(grads []Gradient) error {
if len(grads) == 0 {
log.Info("Send Empty Gradient")
return nil
Copy link
Contributor

@helinwang helinwang Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe return an error. If we treat it as error, user can be aware of this error and fix it.

Btw, let's not use "Title Case" (基本每个单词都大写) for log message. For log message we can use a normal sentence like: "Sent empty gradient." (or people uses "sent empty gradient" (首单词不大写,不用句号结尾))。

For error message, people typically use "sent empty gradient" (首单词不大写,不用句号结尾), because error messages are often combined somewhere else. E.g.,

func foo() error {
  return errors.New("sent empty gradient")
}

func main() {
  err := foo()
  if err != nil {
    log.Errorf("foo returned error: %v", err)
  }
}

// Output:
// foo returned error: sent empty gradient

If we use a normal sentence like: "Sent empty gradient.", the output would be: "foo returned error: Sent empty gradient.", Which could be strange, because in English grammar, word typically does not capitalize after ":".

Also, "send empty gradient" may not express this error fully, "no gradient received" could be better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for such a detailed comment! 👍👍👍
fix Done.

@helinwang
Copy link
Contributor

Thank for noticing this! Good catch!

Copy link
Contributor

@helinwang helinwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM after fixing the only comment.

@dzhwinter dzhwinter merged commit ff4b4cf into PaddlePaddle:develop Jun 29, 2017
heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants