Can a star cool down enough and become a giant planet?
There is a continuum of sorts between very small stars (Red Dwarves) and very large planets (Gas Giants). In between these two, there are the Brown Dwarves, also known as failed stars.
Different parts of the fusion chain, starting with hydrogen and eventually ending with iron, take different amounts of energy to produce.
Brown dwarves are not able to complete the entire fusion chain, and eventually fizzle out.
The precise classification of these objects can vary between sources, but arguably these are the sorts of objects you describe in your question.