-
Notifications
You must be signed in to change notification settings - Fork 74k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Container localhost does not exist. #16481
Comments
I think this could be a knock-on effect from 9f4118d, which modifies (and in principle simplifies) how DT_RESOURCE tensors are captured inside a It sounds like there is some disagreement between the code that creates the lookup table and the op that performs the lookup itself. Could you share the code for creating the table and the /cc @rohan100jain |
@mrry I'd love to share the code but it's written in Scala using my API and it might be hard to get familiar with the codebase. I can do so if you think that can help. However, in order to give you a summary, the lookup table is created right before being used in the dataset map. It is also initialized prior to being used by calling |
Hmm, what you're doing doesn't sound like it's wrong, because that's effectively what the Python version would be doing too. And the recent change should make it less likely to see a Could you possibly create a minimal example that exhibits the problem in both Python and Scala, and share the code for these? If you could also capture the text-format
That's right. Are you using the |
@mrry I can look into creating a minimal example this week, but I have found some information that might be useful. It has to do with creating functions and capturing control dependencies of these functions. Currently, functions are created in a separate graphs and tensors that are used as inputs are captured and replaced with placeholders to be fed later. What about control dependencies that some of the new ops might have on ops defined outside the function? For example, I have a dataset map op that uses a lookup table, but I want the iterator initializer op for that dataset to have a control dependency on the lookup table initializer op of the outer graph. How should I go about handling that? Currently I get an error of the form |
@mrry Actually never mind, even if I work around this initialization issue and manage to run all the initializer ops fine, I still get the same error once I get to invoke |
Note also that if I put a breakpoint in my Scala code, this error:
keeps being printed over and over again indefinitely as if it keeps being thrown from within some loop running in another thread. |
@mrry Given that the lookup table initializer op and the iterator initializer op run without any issues, is there any chance the resource manager is cleared/reset afterwards in between the session runs? Is there any tips you might have on how to debug this sort of error? |
@mrry I'm wondering if this has to do with me using the C API |
@mrry I finally resolved this. I was accidentally setting the shared name of the iterator I was creating. I'm not sure why this caused the problem, but without setting the shared name, everything works fine. Do you know why that could have happened? I'm actually curious. |
Thanks for tracking that down! I think this is a legitimate bug, introduced in 9f4118d. That change modifies most iterators to use the same However, when the Thanks for the promising lead on this bug! We'll try to get a fix in soon. |
@mrry Thanks for tracking this down and fixing it! :) |
Thanks for digging into it and making it much easier to fix! |
Hi @mrry @eaplatanios can you please take a look if the issue #50801 is related to this in any way? |
Hi,
I upgraded from 1.5.0-rc1 to the current master branch and I started receiving the following error:
It's hard to reproduce this error but a summary of the context is that I have a lookup table op inside a dataset map operator and I get this error when I try to execute the corresponding iterator "GetNext" op. I'm looking for information in how to parse and debug this error. I never explicitly set any containers for my variables or lookup tables (i.e., leave them to the default value; an empty string). Were there any changes introduced recently that could result in this error? Note that this happens with my Scala API but not with the Python API and so it may be that I haven't updated something in my code. I just don't really know where to look for this.
Thanks!
The text was updated successfully, but these errors were encountered: