Anti-debugging techniques are used when trying to thwart reverse engineering of software. Two common places where this is seen is in video games to prevent hackers from learning game mechanics and commercial software to stop people from writing key generators.
The most trivial way to check if a debugger is present is by calling IsDebuggerPresent. Internally, IsDebuggerPresent checks a flag in the Process Environment Block (PEB). The address of the PEB can be found in the Thread Information Block, which can be found in the FS register. Most softwares do not soley rely on this method as it can easily be bypassed by jumping over the cmp instruction.
BOOL WINAPI IsDebuggerPresent(void);
When access to the program is limited, it cna be checked remotely with CheckRemoteDebuggerPresent. The function acts as a wrapper to NtQueryInformationProcess, which provides infomration about a specified process. One piece of infomration that can be extracted are the available debug ports. When the number is non-zero, a debugger is attached to the process. This again is easily bypassed and software rarely relies on it to stop reverse engineers.
BOOL WINAPI CheckRemoteDebuggerPresent(
_In_ HANDLE hProcess,
_Inout_ PBOOL pbDebuggerPresent
);
Reads the BeingDebugged field in the Process Environment Block(PEB). One way to determine the address of the PEB is by reading offset 0x30 from the Thread Information Block(TIB), which is obtained from the FS segment register. This is how IsDebuggerPresent works internally. Instead of calling IsDebuggerPresent, some software will manually perform this check. It can be bypassed the same way a call to IsDebuggerPresent is.
The Windows API allows for printing debug statements out to a debugger, and will set an error if a debugger is not found. This is used by attempting to send a message to an attached debugger and checking for errors to determine if one is attached. MSDN Documentation
void WINAPI OutputDebugString( In_opt LPCTSTR lpOutputString );
Instead of using debug specific APIs, if the window name of the debugger is known it can be searched for via FindWindow. Software/malware can search for window titles such as OllyDbg, x64dbg, Soft Ice, etc.
MSDN Documentation
HWND WINAPI FindWindow(
_In_opt_ LPCTSTR lpClassName,
_In_opt_ LPCTSTR lpWindowName
);